Information processing device, information processing system, information processing method, and program

ABSTRACT

A configuration to output section information of an automatic driving section and a manual driving section and display data by which a time to reach each section is confirmable to a wearable terminal is implemented. A mobile device acquires the section information of an automatic driving section and a manual driving section on a traveling route, and estimates the time to reach each section and transmits the estimated time to an information terminal. The information terminal receives the transmission data from the mobile device, and outputs the section information of the automatic driving section and the manual driving section and the display data by which a time to reach each section is confirmable. Moreover, the mobile device determines notification timing of a manual driving recovery request notification on the basis of an arousal level, a position, or the like of a driver and transmits the determined notification timing to the information terminal, and the information terminal executes the manual driving recovery request notification at appropriate notification timing.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing system, an information processing method, and aprogram. More specifically, the present disclosure relates to aninformation processing device, an information processing system, aninformation processing method, and a program for executing processing ofproviding information such as switching timing between automatic drivingand manual driving to a driver of an automatically driving vehicle.

BACKGROUND ART

In recent years, for the purpose of improving safety of vehicletraveling, reducing a load on a driver, and the like, systems forautomatically supporting and controlling traveling of a vehicle withouta driver's operation are being developed. In the automatically drivingvehicle, allowance of execution of a secondary task is expected in thefuture. However, as a road infrastructure for the time being,introduction of an environmental infrastructure as a traveling route isexpected, in which a driver intervention required section whereso-called manual driving is required and a automatically drivablesection are mixed in spots. Therefore, to favorably execute thesecondary task by the driver, section information such as a manualsection and an automatic section on the traveling route needs to beappropriately provided to the driver, and the latest information foreach proceeding section during traveling needs to correspond tosituation change that varies moment to moment. Then, end points of theseautomatic driving sections are not always fixed.

For example, Patent Document 1 (Japanese Patent Application Laid-OpenNo. 2016-139204) discloses a technology of displaying danger levels fora plurality of dangers on a road on which a driver's own vehicle isplanned to travel. Furthermore, for example, Patent Document 2 (JapanesePatent Application Laid-Open No. 2016-090274) discloses a technology ofdisplaying and notifying the driver who is focusing on a mobile terminalthat the driver is in a situation to start manual driving on a screen ofthe mobile terminal in the case of causing the driver to start manualdriving during automatic driving.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2016-139204-   Patent Document 2: Japanese Patent Application Laid-Open No.    2016-090274

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Currently, the momentum to use automatic driving of vehicles in a realworld space is increasing due to the maturation of vehicle surroundingenvironment recognition technologies, the development of map informationof road driving environments, and the like for performing automaticdriving. However, in reality, it is difficult to realize an environmentequivalent to an environment with a closed track, such as a subway, at apractical cost in every continuous section where vehicles can travel.Therefore, in reality, a traveling section where an automatic drivingpermissible section that is a road of a driving level where allautomatic driving is available, and a manual driving section that is aroad where no automatic driving is allowed are randomly connected isassumed. In this way, configuration of a road environment in whichdifferent sections from a section where the automatic driving isavailable to the manual driving section under driver's observation orwhich requires manual driving intervention are mixed is expected. Thenin the case of traveling in such a mixed road section, the vehiclecannot maintain smooth traveling on a route crossing sections ifrecovery to the manual driving required for the driver when crossing thesections is not properly performed.

Therefore, for example, a proposal of using a closed environment wherefull automatic driving is available, and a proposal of urgently stoppingthe vehicle in the case where appropriate takeover from the automaticdriving to the manual driving cannot be performed have been made.However, if an emergency stop vehicle overflows on the road, it maycause a traffic jam, and an increase in the number of emergency stopvehicles may cause accidents that have not happened before. Such asituation impedes normal social activities and is a problem to be solvedfor the spread of automatic driving.

For example, to widely disseminate the level 3 automatic driving, whichis currently widely discussed, to general roads and dedicated roads, anenvironment for enabling level 3 automatic driving vehicles to smoothlytravel in an automatic driving section needs to be constructed andmaintained. Moreover, the driver needs to instantly start normal manualdriving at an end point of the automatic driving section, that is, at anentry point to a manual driving start section.

To enable the driver to instantly start the normal manual driving at theentry point to the manual driving start section, the driver is requiredto always monitor the section and to be in a tense state even in theautomatic driving section. Imposing such a load on the driver is notfavorable in light of ergonomic and psychological aspects. Furthermore,in reality, impaired attention such as the driver falling asleep duringthe automatic driving is well predictable. Furthermore, in aparcel-delivery vehicle or the like, there are some cases where thedriver leaves the driver's seat during the automatic driving and sortsparcels in a luggage compartment, for example. It is assumed that thedriver may forget about approach of a manual driving section during theexecution of such work.

The present disclosure has been made in view of such a realisticsituation, and an object of the present disclosure is to provide aninformation processing device, an information processing system, aninformation processing method, and a program that provide informationsuch as approach of a manual driving start section and the like with adriver of an automatically driving vehicle, and enable the driver tosmoothly start the manual driving at an entry to the manual drivingstart section.

Solutions to Problems

The first aspect of the present disclosure resides in

an information processing device that is a wearable terminal wearable ona human body, the information processing device including:

a display unit configured to output section information of an automaticdriving section and a manual driving section on a traveling route of amobile device, and display data by which a time to reach each section isconfirmable.

Moreover, the second aspect of the present disclosure resides in

an information processing system including a mobile device and aninformation terminal having a wearable specification wearable on adriver of the mobile device, in which

the mobile device

executes processing of acquiring section information of an automaticdriving section and a manual driving section on a traveling route of themobile device, estimating a time to reach each section, and transmittingthe estimated time to the information terminal, and

the information terminal

receives transmission data from the mobile device, and

outputs the section information of an automatic driving section and amanual driving section on a traveling route of the mobile device, anddisplay data by which the time to reach each section is confirmable, toa display unit.

Moreover, the third aspect of the present disclosure resides in

an information processing method executed in an information processingdevice,

the information processing device being a wearable terminal wearable ona human body, the information processing method including:

by a data processing unit, outputting section information of anautomatic driving section and a manual driving section on a travelingroute of a mobile device, and display data by which a time to reach eachsection is confirmable, to a display unit.

Moreover, the fourth aspect of the present disclosure resides in

an information processing method executed in an information processingsystem including a mobile device and an information terminal having awearable specification wearable on a driver of the mobile device, theinformation processing method including:

by the mobile device,

executing processing of acquiring section information of an automaticdriving section and a manual driving section on a traveling route of themobile device, estimating a time to reach each section, and transmittingthe estimated time to the information terminal; and

by the information terminal,

receiving transmission data from the mobile device; and

outputting the section information of an automatic driving section and amanual driving section on a traveling route of the mobile device, anddisplay data by which the time to reach each section is confirmable, toa display unit.

Moreover, the fifth aspect of the present disclosure resides in

a program for causing an information processing device to executeinformation processing,

the information processing device being a wearable terminal wearable ona human body,

the program for causing a data processing unit to output sectioninformation of an automatic driving section and a manual driving sectionon a traveling route of a mobile device, and display data by which atime to reach each section is confirmable, to a display unit.

Note that the program according to the present disclosure is, forexample, a program that can be provided by a storage medium or acommunication medium provided in a computer readable format to aninformation processing device or a computer system that can executevarious program codes. By providing such a program in the computerreadable format, processing according to the program is implemented onthe information processing device or the computer system.

Still other objects, features, and advantages of the present disclosurewill become clear from more detailed description based on examples andattached drawings of the present disclosure to be described below. Notethat a system in the present specification is a logical aggregateconfiguration of a plurality of devices, and is not limited to deviceshaving respective configurations within the same housing.

Effect of the Invention

According to a configuration of an embodiment of the present disclosure,a configuration to output section information of an automatic drivingsection and a manual driving section and display data by which a time toreach each section is confirmable to a wearable terminal is implemented.

Specifically, for example, a mobile device acquires the sectioninformation of the automatic driving section and the manual drivingsection on a traveling route, and estimates the time to reach eachsection and transmits the estimated time to an information terminal. Theinformation terminal receives the transmission data from the mobiledevice, and outputs the section information of the automatic drivingsection and the manual driving section and the display data by which atime to reach each section is confirmable. Moreover, the mobile devicedetermines notification timing of a manual driving recovery requestnotification on the basis of an arousal level, a position of a driver,or the like, and transmits the determined notification timing to theinformation terminal, and the information terminal executes the manualdriving recovery request notification at the notification timing.

With the present configuration, the configuration to output sectioninformation of an automatic driving section and a manual driving sectionand display data by which a time to reach each section is confirmable toa wearable terminal is implemented.

Note that the effects described in the present specification are merelyexamples and are not limited, and additional effects may be exhibited.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing a configuration example of a mobiledevice of the present disclosure.

FIG. 2 is a diagram for describing an example of data displayed on adisplay unit of the mobile device of the present disclosure.

FIG. 3 is diagrams illustrating processing executed by the mobile deviceof the present disclosure.

FIG. 4 is a diagram for describing a use configuration example of aninformation terminal of the present disclosure.

FIG. 5 is a diagram for describing a use configuration example of theinformation terminal of the present disclosure.

FIG. 6 is a diagram for describing a use configuration example of theinformation terminal of the present disclosure.

FIG. 7 is a diagram for describing a use configuration example of theinformation terminal of the present disclosure.

FIG. 8 is a diagram for describing a configuration example of the mobiledevice of the present disclosure.

FIG. 9 is a diagram for describing a configuration example of the mobiledevice according to the present disclosure.

FIG. 10 is a diagram for describing a sensor configuration example ofthe mobile device and the information terminal of the presentdisclosure.

FIG. 11 is a diagram illustrating an example of a mode switchingsequence from an automatic driving mode to a manual driving modeexecuted by the mobile device of the present disclosure.

FIG. 12 is a diagram illustrating an example of a mode switchingsequence from an automatic driving mode to a manual driving modeexecuted by the mobile device of the present disclosure.

FIG. 13 is a diagram illustrating a flowchart for describing an exampleof an operation sequence of automatic driving.

FIG. 14 is a diagram illustrating an example of a traveling route inwhich automatic driving available and unavailable sections determined asa driver sets a destination are set or determined in spots.

FIG. 15 is a diagram for describing an example of display data of theinformation terminal of the present disclosure.

FIG. 16 is a diagram for describing an example of the display data ofthe information terminal of the present disclosure.

FIG. 17 is a diagram for describing an example of the display data ofthe information terminal of the present disclosure.

FIG. 18 is a diagram for describing an example of the display data ofthe information terminal of the present disclosure.

FIG. 19 is a diagram illustrating a flowchart for describing an exampleof a sequence of processing executed when switching the automaticdriving mode to the manual driving mode.

FIG. 20 is graphs for describing a distribution example of a pluralityof pieces of relationship information (observation plots) between anobservable evaluation value corresponding to an observation value and arecovery delay time (=manual driving recoverable time), and a recoveryratio.

FIG. 21 is a graph for describing a manual driving recoverable timeaccording to a type of processing (secondary task) executed by a driverin the automatic driving mode.

FIG. 22 is a diagram for describing a hardware configuration example ofan information processing device.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an information processing device, an information processingsystem, and an information processing method, and a program of thepresent disclosure will be described in detail with reference to thedrawings. Note that the description will be given according to thefollowing items.

1. Outline of Configurations and Processing of Mobile Device andInformation Processing Device (Information Terminal)

2. Specific Configuration and Processing Example of Mobile Device

3. Mode Switching Sequence from Automatic Driving Mode to Manual DrivingMode

4. Operation Sequence Example of Automatic Driving

5. Sequence of Processing Executed by Mobile Device and InformationProcessing Device (Information Terminal) of Present Disclosure

6. Configuration Example of Information Processing Device

7. Conclusion of Configurations of Present Disclosure

[1. Outline of Configurations and Processing of Mobile Device andInformation Processing Device (Information Terminal)]

An outline of configurations and processing of a mobile device and aninformation processing device (information terminal) will be describedwith reference to FIG. 1 and the subsequent drawings.

The mobile device is an automobile capable of traveling while switchingthe automatic driving and the manual driving, for example. In such anautomobile, in a case where necessity to switch the automatic drivingmode to the manual driving mode occurs, it is necessary to cause adriver to start the manual driving.

However, there are various types of processing (secondary tasks)performed by the driver during execution of the automatic driving. Forexample, there are a case where the driver is looking at the front ofthe vehicle while the driver is merely releasing a steering wheel,similarly to the case of driving, a case where the driver is reading abook, and a case where the driver falls asleep. The arousal level(consciousness level) of the driver varies depending on the differencein these types of processing.

For example, falling asleep reduces the driver's arousal level. That is,the consciousness level is lowered. In such a state where the arousallevel is lowered, normal manual driving cannot be performed, and if themode is switched to the manual driving mode in the state, an accidentmay occur in the worst case.

To secure the safety of driving, it is necessary to cause the driver tostart the manual driving in a state where the driver's arousal level ishigh, that is, in a state where the driver is clearly conscious.Therefore, it is necessary to change the notification timing forrequesting switching from the automatic driving to the manual drivingaccording to the arousal level of the driver who is executing theautomatic driving.

For example, in a case where the driver looks forward and is looking atthe road while executing the automatic driving, the driver's arousallevel is high, that is, the driver can start the manual driving at anytime. In such a case, notification for switching to the manual drivingis simply issued immediately before the time when the manual driving isrequired. This is because the driver can immediately start safe manualdriving.

However, in a case where the driver falls asleep during execution of theautomatic driving, the driver's arousal level is extremely low. In sucha case, if notification for switching to the manual driving is issued attiming immediately before the time when the manual driving is required,the driver is forced to start the manual driving in the state whereconsciousness is unclear. As a result, the possibility of causing anaccident increases. Therefore, in the case where the arousal level islow, as described above, it is necessary to issue notification of theswitching to the manual driving at an earlier stage.

Furthermore, an automobile 10 illustrated in FIG. 1 is a wagon vehiclefor parcel delivery, and a driver 20 sometimes leaves a driver's seatduring the automatic driving and moves to a luggage compartment 21 andsorts the parcels for deliver, for example. As described above, in thecase where the driver 20 is away from the driver's seat, it is necessaryto control manual driving switching notification timing in considerationof time to return to the driver's seat.

As described above, at present, the infrastructure for enabling theautomatic driving in all of traveling sections has not been realized,and there are sections where the manual driving is required. Therefore,the driver is required to recover from the automatic driving to themanual driving at various timings and points. The configuration of thepresent disclosure provides approach information of a manual drivingstart section and the like to the driver in an automatic drivingvehicle, and enables the driver to smoothly start the manual driving atthe time of entering the manual driving start section.

An outline of configurations and processing of a mobile device and aninformation processing device (information terminal 50 illustrated inFIG. 1) will be described with reference to FIG. 1 and the subsequentdrawings.

FIG. 1 illustrates a configuration example of the automobile 10 that isan example of the mobile device.

Furthermore, the information terminal 50 of the present disclosure isattached to an arm of the driver 20 in the automobile 10 illustrated inFIG. 1.

The information terminal 50 is a wearable terminal wearable on a humanbody. Here, as an example of the wearable terminal, a wristwatch-typeterminal is illustrated. An information processing device of the presentdisclosure is not limited to such a wristwatch-type terminal, and may bea wearable terminal attachable to various portions of a person, such asa terminal worn on the head, a terminal worn on a waist, or a ring-typeterminal, for example.

The automobile 10 illustrated in FIG. 1 is an automobile capable ofdriving in two driving modes of the manual driving mode and theautomatic driving mode.

In the manual driving mode, traveling based on an operation of a driver20, that is, an operation of a steering wheel (steering), an operationof an accelerator, a brake, or the like is performed.

Meanwhile, in the automatic driving mode, the operation by the driver 20is unnecessary, and driving based on sensor information such as aposition sensor and other ambient information detection sensors isperformed.

The position sensor is, for example, a GPS receiver or the like, and theambient information detection sensor is, for example, a camera, anultrasonic sensor, a radar, a light detection and ranging or a laserimaging detection and ranging (LiDAR), a sonar, or the like.

Note that FIG. 1 is a diagram for describing an outline of the presentdisclosure and schematically illustrates main configuration elements.Detailed configurations will be described below.

As illustrated in FIG. 1, the automobile 10 includes a data processingunit 11, a driver information acquisition unit 12, an environmentinformation acquisition unit 13, a communication unit 14, and anotification unit 15.

The driver information acquisition unit 12 acquires, for example,information for determining the arousal level of the driver, such asbiometric information of the driver, and operation information of thedriver. Specifically, for example, the driver information acquisitionunit 12 includes a camera that captures a face image of the driver, asensor that acquires motions of eyeballs and pupils or the like, ameasurement sensor for temperature or the like, and an operationinformation acquisition unit for the operation units (steering wheel,accelerator, brake, and the like), and the like.

The environment information acquisition unit 13 acquires travelingenvironment information of the automobile 10. For example, imageinformation of the front, rear, right, and left of the automobile, andsurrounding obstacle information from the light detection and ranging orthe laser imaging detection and ranging (LiDAR), the sonar, or the like.

The data processing unit 11 receives the driver information acquired bythe driver information acquisition unit 12 and the environmentinformation acquired by the environment information acquisition unit 13as inputs, and calculates safety index values indicating whether or notthe driver in the automatic driving vehicle is in a safe manual drivingexecutable state, and moreover, whether or not the driver in the manualdriving is executing safe driving, for example.

Moreover, for example, in the case where necessity of switching from theautomatic driving mode to the manual driving mode arises, the dataprocessing unit 11 executes processing of issuing notification forswitching to the manual driving mode via the notification unit 15.

This notification processing timing is optimum timing calculated usingthe inputs from the driver information acquisition unit 12 and theenvironment information acquisition unit 13, for example.

That is, it is the timing when the driver 20 can start safe manualdriving.

Specifically, in the case where the arousal level of the driver is high,the notification is issued immediately before the manual driving starttime, for example, five seconds before. In the case where the arousallevel of the driver is low, the notification is issued twenty secondsbefore the manual driving start time with a margin, for example.Specific calculation of the optimum timing for the notification will bedescribed below.

The notification unit 15 includes a display unit that displays thenotification, a sound output unit, a steering wheel, or a vibrator of aseat. An example of warning display displayed on the display unitconstituting the notification unit 15 is illustrated in FIG. 2.

As illustrated in FIG. 2, the notification unit (display unit) 15displays the following items.

Driving mode information=“In automatic driving”,

Warning display=“Please switch driving to manual driving”

“In automatic driving” is displayed at the time of executing theautomatic driving mode, and “In manual driving” is displayed at the timeof executing the manual driving mode, in a display area of the drivingmode information.

The display area of the warning display information is a display areawhere the following item is displayed while the automatic driving isexecuted in the automatic driving mode.

“Please switch driving to manual driving”

Note that the automobile 10 has a configuration capable of communicatingwith a server 30 via the communication unit 14, as illustrated in FIG.1.

For example, part of processing of calculating appropriate time of anotification output in the data processing unit 11 can be performed bythe server 30.

A specific example of the processing will be described below.

FIG. 3 is a diagram illustrating a specific example of processingexecuted by the mobile device and the information processing device ofthe present disclosure.

FIG. 3 is a diagram illustrating a setting example of appropriate timingof the notification for requesting switching to the manual driving whilethe automatic driving is executed in the automatic driving mode, andillustrates the following two notification processing examples.

(a) Notification processing in the case where the driver's arousal levelduring the automatic driving is high

(b) Notification processing in the case where the driver's arousal levelduring the automatic driving is low

(c) Notification processing in the case where the driver who isexecuting the automatic driving is away from the driver's seat

The example (a) is an example in which the driver is looking at the roadin front while executing the automatic driving. In this case, thedriver's arousal level is high, that is, the user can start the manualdriving at any time.

In such a case, even if the notification for switching to the manualdriving is issued at timing immediately before the time when the manualdriving is required, the driver can start safe manual driving soon.

The example (b) is an example in which the driver falls asleep whileexecuting the automatic driving, and in this case, the driver's arousallevel is extremely low.

In such a case, if the notification for switching to the manual drivingis issued at timing immediately before the time when the manual drivingis required, the driver starts the manual driving in the state whereconsciousness is unclear, and there is a high possibility of causing anaccident. Therefore, in the case where the arousal level is low, asdescribed above, it is necessary to issue the notification of theswitching to the manual driving at an earlier stage.

The example in (c) is an example in which the driver is away from thedriver's seat and working during the automatic driving. In such a case,it takes time for the driver to return to the driver's seat.

In such a case, if notification for switching to the manual driving isissued at timing immediately before the time when the manual driving isrequired, there is a possibility that the vehicle approaches the manualdriving section before the driver returns the driver's seat. Therefore,in the case where the driver is away from the driver's seat, it isnecessary to issue notification of the switching to the manual drivingat an earlier stage.

Furthermore, in the case where the driver 20 in the automobile 10illustrated in FIG. 1 is away from the driver's seat, the driver 20cannot notice display as described with referred to FIG. 2 on anotification unit (display unit) 15 at the driver's seat.

To solve the problem, information or notification is provided via theinformation terminal 50 worn on an arm of the driver 20. The informationterminal 50 performs display processing for the display data describedwith reference to FIG. 2, and further displays approach information ofthe section (automatic driving section or manual driving section) on theroad on which the automobile 10 is traveling, and the like. Moreover,the information terminal 50 has an alarm output, a voice output, and avibration function, and provides the driver 20 with variousnotifications and alarms.

FIG. 4 is a diagram illustrating a use example of the informationterminal 50. There are some cases where the driver 20 in the automobile10 sits in the driver's seat as illustrated in FIG. 4(a) or leave thedriver's seat and work in the luggage compartment, as illustrated inFIG. 4(b), while the automobile 10 is executing the automatic driving.However, the driver 20 always wears the information terminal 50 onhis/her arm and can notice the display information displayed on theinformation terminal 50, an output sound, an alarm, a vibration, and thelike.

FIG. 5 is a diagram illustrating an example of the display informationon the information terminal 50. The example in FIG. 5 illustrates adisplay example similar to the display example of the notification unit(display unit) 15 described above with reference to FIG. 2. Thefollowing displays are made on the information terminal 50.

Driving mode information=“In automatic driving”,

Warning display=“Please switch driving to manual driving”

The driver 20 can check this display anywhere.

Moreover, FIG. 6 illustrates another example of the display informationon the information terminal 50. The example illustrated in FIG. 6 is anexample of displaying information of sections such as the automaticdriving section and the manual driving section on the road according toa traveling route determined as the driver sets a destination. Thenumerical values 0 to 9 around the display unit indicate elapsed time of0 to 9 minutes from the current time. The entire display area shows thesection information of the scheduled traveling of the automobile fromthe current time (0) to 10 minutes later.

The driving route is provided with an automatic driving section Sa and amanual driving section Sb, and further, a takeover section Sc from theautomatic driving to the manual driving, a cautioned traveling sectionSd set in the automatic driving section Sa, and the like. Note thatspecific examples of these section settings will be described below.

In the display example illustrated in FIG. 6, the following threedifferent display areas are set on the display unit of the informationterminal 50.

(Sa) Automatic driving section Sa (=green display)

(Sc) Takeover driving section Sc (=yellow display)

(Sb) Manual driving section Sb (=red display)

The display example in FIG. 6 illustrates the traveling schedule of theautomobile for 10 minutes from a present time. That is, the automobileis scheduled to travel

in (Sa) automatic driving section Sa (=green display) after from 0 to 6minutes and 10 seconds from the present time, then

in (Sc) takeover driving section Sc (=yellow display) after from 6minutes and 10 seconds to 8 minutes and 40 seconds from the presenttime, and

further, the automobile is scheduled to travel

in (Sb) manual driving section Sb (=red display) after 8 minutes and 40seconds from the present time.

Note that the information terminal 50 receives, via the communicationunit, scheduled traveling time information of each section calculated bythe data processing unit 11 in the automobile 10 on the basis of atraveling speed of the automobile 10 or average speed information of thevehicle on the traveling route of the automobile 10, and displays thereceived information.

The driver 20 can confirm a remaining time to enter the manual drivingsection by seeing the display information on the information terminal50, and can finish the work in the luggage compartment 21 and return tothe driver's seat in time.

Note that, in the example illustrated in FIG. 6, the driver 20 can graspentry to the manual driving section about after 8 minutes and 40 secondsfrom the present time on the basis of the display on the informationterminal 50, can proceeds with the work to return to the driver's seattill then.

Furthermore, an arrow is further displayed on the information terminal50 illustrated in FIG. 6. This arrow is set to the time of notifying thedriver 20 of the notification of the manual driving recovery request,for example, switching to the display illustrated in FIG. 5, oroutputting an alarm or starting a vibration to warn the driver 20. Thisnotification timing is determined on the basis of information of atleast either the arousal level or the position of the driver on themobile device (automobile 10) side, for example. The determinednotification timing information is transmitted to the informationterminal 50. The time display of the notification using the arrow isonly an example, and it is not necessary to limit the means to the arrowas long as the time interval can be visually presented, for example, byproviding a visual attention attracting function by colorclassification, mesh display, blinking flash, or the like. Note thatprocessing of calculating the optimum notification timing executed bythe mobile device will be described below.

The information terminal 50 executes the manual driving recovery requestnotification at the notification timing received from the mobile device.That is, the information terminal 50 executes at least one of thedisplay processing, the alarm output, or the vibration activationdescribed with reference to FIG. 5 at the notification timing determinedby the mobile device. Note that the notification setting can be changedor stopped by the user.

Note that, in the present embodiment, the display unit of theinformation terminal 50 is illustrated as a circular display unit, butthis is an example. The shape of the display unit is not limited to acircle, and various shapes such as an elliptical shape and a rectangularshape can be set.

FIG. 7 is a diagram illustrating an example of changes in display dataon the information terminal with passage of time. FIG. 7 illustrates anexample of the display information on the information terminal 50 attime t1 and the display information on the information terminal 50 atsubsequent time t2.

The time t2 shows an example of the display data about 4 minutes afterthe time t1.

The display data at the time t2 indicates that the manual drivingsection Sb (=red display) is scheduled to travel after about 3 minutesand 50 seconds from the present time.

At the time t2, the driver 20 grasps that the manual driving section Sb(=red display) is approaching on the basis of the display on theinformation terminal 50, quickly finishes the work, and can startpreparation for returning to the driver's seat.

Note that the information terminal 50 has an alarm output function and avibration function, and for example, when the remaining time to themanual driving section Sb (=red display) reaches a predefined time (forexample, 1 minute), the information terminal 50 outputs an alarm andactivates vibration to warn the driver 20. Note that the activation timeof the alarm and vibration can be changed or stopped according to usersettings.

[2. Specific Configuration and Processing Example of Mobile Device]

Next, a specific configuration and a processing example of a mobiledevice corresponding to the automobile 10 of the present disclosure willbe described with reference to FIG. 8 and the subsequent drawings.

FIG. 8 illustrates a configuration example of the mobile device 100.Note that, hereinafter, in a case of distinguishing a vehicle providedwith the mobile device 100 from other vehicles, the vehicle is referredto as user's own car or user's own vehicle.

The mobile device 100 includes an input unit 101, a data acquisitionunit 102, a communication unit 103, an in-vehicle device 104, an outputcontrol unit 105, an output unit 106, a drive system control unit 107, adrive system 108, a body system control unit 109, a body system 110, astorage unit 111, and an automatic driving control unit 112.

The input unit 101, the data acquisition unit 102, the communicationunit 103, the output control unit 105, the drive system control unit107, the body system control unit 109, the storage unit 111, and theautomatic driving control unit 112 are connected to one another via acommunication network 121. The communication network 121 includes, forexample, an on-board communication network conforming to an arbitrarystandard such as a controller area network (CAN), a local interconnectnetwork (LIN), a local area network (LAN), or FlexRay (registeredtrademark), a bus, and the like. Note that the units of the mobiledevice 100 may be directly connected without the communication network121.

Note that, hereinafter, the case where the units of the mobile device100 perform communication via the communication network 121, thedescription of the communication network 121 is omitted. For example,the case where the input unit 101 and the automatic driving control unit112 perform communication via the communication network 121 will besimply described as the input unit 101 and the automatic driving controlunit 112 performing communication.

The input unit 101 includes a device used by a passenger to inputvarious data and instructions. For example, the input unit 101 includesoperation devices such as a touch panel, a button, a microphone, aswitch, and a lever, and an operation device capable of inputting dataand instructions by a method other than a manual operation, such asvoice or gesture. Furthermore, for example, the input unit 101 may be aremote control device using infrared rays or other radio waves, or anexternally connected device such as a mobile device or a wearable devicecorresponding to the operation of the mobile device 100. The input unit101 generates an input signal on the basis of the data, instructions,and the like input by the passenger, and supplies the input signal toeach unit of the mobile device 100.

The data acquisition unit 102 includes various sensors that acquire datato be used for the processing of the mobile device 100, and supplies theacquired data to each unit of the mobile device 100.

For example, the data acquisition unit 102 includes various sensors fordetecting the state of the user's own car. Specifically, for example,the data acquisition unit 102 is a gyro sensor, an acceleration sensor,an inertial measurement device (IMU), and sensors for detecting anoperation amount of an accelerator pedal, an operation amount of a brakepedal, a steering angle of a steering wheel, an engine speed, a motorspeed, a rotation speed of wheels, or the like.

Furthermore, for example, the data acquisition unit 102 includes varioussensors for detecting information outside the user's own car.Specifically, for example, the data acquisition unit 102 includesimaging devices such as a time of flight (ToF) camera, a stereo camera,a monocular camera, an infrared camera, and other cameras. Furthermore,for example, the data acquisition unit 102 includes an environmentsensor for detecting a weather, a meteorological phenomenon, or thelike, and an ambient information detection sensor for detecting anobject around the user's own car. The environment sensor includes, forexample, a raindrop sensor, a fog sensor, a sunshine sensor, a snowsensor, and the like. The ambient information detection sensor includes,for example, an ultrasonic sensor, a radar device, a light detection andranging or laser imaging detection and ranging (LiDAR) device, or asonar.

For example, FIG. 9 illustrates an installation example of the varioussensors for detecting external information of the user's own car. Eachof imaging devices 7910, 7912, 7914, 7916, and 7918 is provided at atleast one position of a front nose, side mirrors, a rear bumper, a backdoor, or an upper portion of a windshield in an interior of a vehicle7900, for example.

The imaging device 7910 provided at the front nose and the imagingdevice 7918 provided at an upper portion of the windshield in aninterior of the vehicle mainly acquire front images of the vehicle 7900.The imaging devices 7912 and 7914 provided at the side mirrors mainlyacquire side images of the vehicle 7900. The imaging device 7916provided at the rear bumper or the back door mainly acquires a rearimage of the vehicle 7900. The imaging device 7918 provided at the upperportion of the windshield in the interior of the vehicle is mainly usedfor detecting a preceding vehicle, a pedestrian, an obstacle, a trafficsignal, a traffic sign, a lane, or the like. Furthermore, in the futureautomatic driving, when the vehicle turns right or left, the imagingdevices may be used in an extended manner up to pedestrians crossing aroad beyond the right or left-turn road in a wider range or an objectrange near a crossing road when the vehicle turns right or left.

Note that FIG. 9 illustrates an example of capture ranges of the imagingdevices 7910, 7912, 7914, and 7916. An imaging range a indicates animaging range of the imaging device 7910 provided at the front nose,imaging ranges b and c respectively indicate imaging ranges of theimaging devices 7912 and 7914 provided at the side mirrors, and animaging range d indicates an imaging range of the imaging device 7916provided at the rear bumper or the back door. For example, a bird's-eyeview image of the vehicle 7900 as viewed from above, an all-roundstereoscopic display image surrounding a vehicle periphery with a curvedplane, and the like can be obtained by superimposing image data imagedin the imaging devices 7910, 7912, 7914, and 7916.

Sensors 7920, 7922, 7924, 7926, 7928, and 7930 provided at the front,rear, side, corner, and upper portion of the windshield in the interiorof the vehicle 7900 may be ultrasonic sensors or radars, for example.Sensors 7920, 7926, and 7930 provided at the front nose, the rearbumper, the back door, and the upper portion of the windshield in theinterior of the vehicle 7900 may be an LiDAR, for example. These sensors7920 to 7930 are mainly used for detecting a preceding vehicle, apedestrian, an obstacle, and the like. Results of the detections may befurther applied to improvement of stereoscopic object display of thebird's-eye view display and the all-round stereoscopic display.

Description of the configuration elements will be continued returning toFIG. 8. The data acquisition unit 102 includes various sensors fordetecting a current position of the user's own car. Specifically, forexample, the data acquisition unit 102 includes a global navigationsatellite system (GNSS) receiver that receives a GNSS signal from a GNSSsatellite.

Furthermore, for example, the data acquisition unit 102 includes varioussensors for detecting information inside the vehicle. Specifically, forexample, the data acquisition unit 102 includes an imaging device thatimages a driver, a biosensor that detects biometric information of thedriver, a microphone that collects sound in a vehicle interior, and thelike. The biosensor is provided on, for example, a seating surface, asteering wheel, or the like, and detects a sitting state of an occupantsitting on a seat or biometric information of the driver holding thesteering wheel. As a vital signal, diversified observable data isavailable such as heart rate, pulse rate, blood flow, respiration,mind-body correlation, visual stimulation, EEG, sweating state, headposture behavior, eye, gaze, blink, saccade, microsaccade, fixation,drift, gaze, and iris pupil reaction. These activity observableinformation reflecting an observable driving state is aggregated asobservable evaluation values estimated from observations, and recoverydelay time characteristics associated with logs of the evaluation valuesare used as specific characteristics to a recovery delay case of thedriver for calculating the recovery notification timing by a safetydetermination unit (learning processing unit) 155 to be described below.

FIG. 10 illustrates an example of various sensors for obtaininginformation of the driver inside the vehicle included in the dataacquisition unit 102. Note that the data acquisition unit 102corresponds to the driver information acquisition unit 12 of theautomobile 10 illustrated in FIG. 1.

Moreover, FIG. 10 illustrates an example of a sensor mounted in theinformation terminal 50.

For example, the data acquisition unit 102 includes a ToF camera, astereo camera, a seat strain gauge, and the like as detectors fordetecting the position and posture of the driver. Furthermore, the dataacquisition unit 102 includes a face recognition device (face (head)recognition), a driver eye tracker, a driver head tracker, and the like,as detectors for obtaining the activity observable information of thedriver.

Furthermore, the data acquisition unit 102 includes a vital signaldetector as a detector for obtaining activity observable information ofthe driver. Furthermore, the data acquisition unit 102 includes a driverauthentication (driver identification) unit. Note that, as anauthentication method, biometric authentication using a face, afingerprint, an iris of a pupil, a voiceprint, or the like can beconsidered in addition to knowledge authentication using a password, apersonal identification number, or the like.

The information terminal 50 includes a position sensor such as a GPSsensor as a detector for detecting the position of the driver.Furthermore, the information terminal 50 includes a vital signaldetector, that is, a biosensor or an activity amount measurement sensor,as a detector for obtaining activity observable information of thedriver. These sensors detect the state of the driver 20 wearing theinformation terminal 50, for example, motion, position, activity amount,activity history, and the like of the driver 20. Furthermore, theinformation terminal 50 includes a driver authentication (driveridentification) unit. Note that, as an authentication method, forexample, biometric authentication using a face, a fingerprint, an irisof a pupil, a voiceprint, or the like is used, in addition to knowledgeauthentication using a password, a personal identification number, orthe like.

Note that the data acquisition unit 102 can acquire a vital signal, anactivity situation, and the like of the driver from the wearableterminal that is the information terminal 50. Furthermore, the driver'spassive state monitoring and active response reaction can be acquired byincorporating a driver's reflective response to the notification to theinformation terminal 50 from the system on the automobile side, forexample. Furthermore, in the case where the driver wears the informationterminal 50 before getting in the vehicle, log data indicating thedriver's state before getting in the vehicle acquired by the informationterminal 50 is stored in the storage unit in the information terminal50, and the log data stored in the storage unit is output to the systemon the vehicle side after the driver gets in the vehicle. The system onthe vehicle side can estimate an arousal state or a functional state ofthe driver, using the log data indicating the driver's state beforeboarding input from the information terminal 50.

Note that the detection information of the information terminal 50 istransmitted to the automobile 10 via the communication unit and storedin the storage unit of the automobile 10. Alternatively, the detectioninformation is transmitted to the server 30 and stored in the storageunit of the server 30. These data are used as learning data to beapplied to time estimation processing required for the driver 20 torecover to the manual driving.

The communication unit 103 communicates with the in-vehicle device 104and various devices outside the vehicle, a server, a base station, andthe like, transmits data supplied from each unit of the mobile device100, and supplies received data to each unit of the mobile device 100.Note that a communication protocol supported by the communication unit103 is not especially limited, and the communication unit 103 cansupport a plurality of types of communication protocols.

For example, the communication unit 103 performs wireless communicationwith the in-vehicle device 104, using a wireless LAN, Bluetooth(registered trademark), near field communication (NFC), a wireless USB(WUSB), or the like. Furthermore, for example, the communication unit103 performs wired communication with the in-vehicle device 104, using auniversal serial bus (USB), a high-definition multimedia interface(HDMI) (registered trademark), mobile high-definition link (MHL), or thelike via a connection terminal (not illustrated) (and a cable ifnecessary).

Moreover, for example, the communication unit 103 communicates with adevice (for example, an application server or a control server) existingon an external network (for example, the Internet, a cloud network, or acompany specific network) via a base station or an access point.Furthermore, for example, the communication unit 103 communicates with aterminal (for example, a terminal of a pedestrian or a shop, or amachine type communication (MTC) terminal) existing in the vicinity ofthe user's own car, using a peer to peer (P2P) technology.

Moreover, for example, the communication unit 103 performs V2Xcommunication such as vehicle to vehicle communication, vehicle toinfrastructure communication, vehicle to home communication, and vehicleto pedestrian communication. Furthermore, for example, the communicationunit 103 includes a beacon reception unit, and receives a radio wave oran electromagnetic wave transmitted from a wireless station or the likeinstalled on a road, and acquires information such as a currentposition, congestion, traffic regulation, or required time. Note thatpairing may be made with a vehicle traveling ahead while traveling in asection, which can be a leading vehicle, through the communication unit,and information acquired by a data acquisition unit mounted on thevehicle ahead may be acquired as pre-travel information and may becomplementarily used as the data of the data acquisition unit 102 of theuser's own car. In particular, this will be a means to secure the safetyof following platooning vehicles, using platooning travel by the leadingvehicle, for example.

The in-vehicle device 104 includes, for example, a mobile device (atablet, a smartphone, or the like) or a wearable device of a passenger,an information device carried in or attached to the user's own car, anda navigation device for searching for a route to an arbitrarydestination. Note that, considering that an occupant is not always fixedat a seat fixing position due to the spread of the automatic driving,the in-vehicle device 104 may be expanded to a video player, a gamedevice, or any other devices that can be installed and removed from thevehicle in the future. In the present embodiment, an example in whichpresentation of information of points requiring intervention of thedriver is limited to an appropriate driver has been described. However,the information may be further provided to a subsequent vehicle inplatooning traveling or the like, or the information provision may becombined with remote travel support by constantly providing theinformation to an operation management center of passengertransportation shared buses and long-distance logistics commercialvehicles, as appropriate.

The output control unit 105 controls output of various types ofinformation to the passenger of the user's own car or to the outside ofthe vehicle. The output control unit 105 controls output of visualinformation (for example, image data) and auditory information (forexample, sound data) from the output unit 106 by generating an outputsignal including at least one of the visual information or the auditoryinformation and supplying the output signal to the output unit 106, forexample. Specifically, for example, the output control unit 105synthesizes image data captured by different imaging devices of the dataacquisition unit 102 to generate a bird's-eye view image, a panoramicimage, or the like, and supplies an output signal including thegenerated image to the output unit 106. Furthermore, for example, theoutput control unit 105 generates sound data including a warning sound,a warning message, or the like for dangers of collision, contact, entryto a dangerous zone, or the like and supplies an output signal includingthe generated sound data to the output unit 106.

The output unit 106 includes a device capable of outputting the visualinformation or the auditory information to the passenger of the user'sown car or to the outside of the vehicle. For example, the output unit106 includes a display device, an instrument panel, an audio speaker,headphones, a wearable device such as a glasses-type display worn by thepassenger, a projector, a lamp, or the like. The display device includedin the output unit 106 may be, for example, a head-up display, atransmission-type display, or a display for displaying the visualinformation in a field of view of the driver, such as a device having anaugmented reality (AR) display function, in addition to a device havinga normal display.

The drive system control unit 107 controls the drive system 108 bygenerating various control signals and supplying the control signals tothe drive system 108. Furthermore, the drive system control unit 107supplies a control signal to each unit other than the drive system 108to issue notification of a control state of the drive system 108, or thelike, as needed.

The drive system 108 includes various devices related to the drivesystem of the user's own car. For example, the drive system 108 includesa drive force generation device for generating a drive force of aninternal combustion engine or a drive motor, a drive force transmissionmechanism for transmitting the drive force to the wheels, a steeringmechanism for adjusting the steering angle, a braking device forgenerating a braking force, an antilock brake system (ABS), anelectronic stability control (ESC), an electric power steering device,and the like.

The body system control unit 109 controls the body system 110 bygenerating various control signals and supplying the control signals tothe body system 110. Furthermore, the body system control unit 109supplies a control signal to each unit other than the body system 110and notifies a control state of the body system 110, or the like, asneeded.

The body system 110 includes various body-system devices mounted on avehicle body. For example, the body system 110 includes a keyless entrysystem, a smart key system, a power window device, a power seat, asteering wheel, an air conditioner, various lamps (for example,headlights, backlights, brake lights, blinkers, fog lights, and thelike), and the like.

The storage unit 111 includes, for example, a magnetic storage devicesuch as a read only memory (ROM), a random access memory (RAM), and ahard disc drive (HDD), a semiconductor storage device, an opticalstorage device, a magneto-optical storage device, and the like. Thestorage unit 111 stores various programs, data, and the like used byeach unit of the mobile device 100. For example, the storage unit 111stores map data such as a three-dimensional high-precision map such as adynamic map, a global map having less accuracy than the high-precisionmap but covering a large area, and a local map including informationaround the user's own car.

The automatic driving control unit 112 performs control related to theautomatic driving such as autonomous driving or driving support.Specifically, for example, the automatic driving control unit 112performs cooperative control for the purpose of implementing an advanceddriver support system (ADAS) function including collision avoidance orshock mitigation of the user's own car, following travel based on avehicular gap, vehicle speed maintaining travel, collision warning ofthe user's own car, lane out warning of the user's own car, and thelike. Furthermore, for example, the automatic driving control unit 112performs the cooperative control for the purpose of automatic driving ofautonomous travel without depending on an operation of the driver. Theautomatic driving control unit 112 includes a detection unit 131, aself-position estimation unit 132, a situation analysis unit 133, aplanning unit 134, and an operation control unit 135.

The detection unit 131 detects various types of information necessaryfor controlling the automatic driving. The detection unit 131 includes avehicle exterior information detection unit 141, a vehicle interiorinformation detection unit 142, and a vehicle state detection unit 143.

The vehicle exterior information detection unit 141 performs processingof detecting information outside the user's own car on the basis of dataor signals from each unit of the mobile device 100. For example, thevehicle exterior information detection unit 141 performs detectionprocessing, recognition processing, and tracking processing, for anobject around the user's own car, and processing of detecting a distanceto the object and a relative speed. Objects to be detected include, forexample, vehicles, people, obstacles, structures, roads, traffic lights,traffic signs, road markings, and the like.

Furthermore, for example, the vehicle exterior information detectionunit 141 performs processing of detecting an environment around theuser's own car. The surrounding environment to be detected includes, forexample, weather, temperature, humidity, brightness, road surfacecondition, and the like. The vehicle exterior information detection unit141 supplies data indicating results of the detection processing to theself-position estimation unit 132, a map analysis unit 151, a trafficrule recognition unit 152, and a situation recognition unit 153 of thesituation analysis unit 133, and an emergency avoidance unit 171 and thelike of the operation control unit 135.

The information acquired by the vehicle exterior information detectionunit 141 can be mainly supplied and received from an infrastructure inthe case of a section stored in the local dynamic map, the section beingconstantly and importantly updated as a section where traveling by theautomatic driving is available. Alternatively, the user's own vehiclemay travel by constantly receiving information update in advance beforeentering a section, from a vehicle or a vehicle group traveling ahead inthe section. Furthermore, in particular, for the purpose of more safelyobtaining road information immediately before entering a section in aplatooning travel, such as a case where the latest local dynamic map isnot constantly updated by the infrastructure, road environmentinformation obtained from a leading vehicle having entered the sectionmay be further supplementarily used. In many cases, the section wherethe automatic driving is available depends on the presence or absence ofprior information provided by these infrastructures. The informationregarding availability of automatic driving on a route provided by aninfrastructure is equivalent to providing an unseen track as so-called“information”. Note that the vehicle exterior information detection unit141 is illustrated on the assumption that the vehicle exteriorinformation detection unit 141 is mounted on the user's own vehicle forthe sake of convenience. Pre-predictability at the time of traveling maybe further improved by using information captured by a preceding vehicleas “information”.

The vehicle interior information detection unit 142 performs processingof detecting information inside the vehicle on the basis of data orsignals from each unit of the mobile device 100. For example, thevehicle interior information detection unit 142 performs driverauthentication processing and recognition processing, driver statedetection processing, passenger detection processing, vehicle interiorenvironment detection processing, and the like. The state of the driverto be detected includes, for example, physical condition, arousal level,concentration level, fatigue level, line-of-sight direction, detailedeyeball behavior, and the like.

Moreover, in the future, the driver is expected to completely taking thedriver's hands off from driving and steering operation in the automaticdriving, and the driver temporarily goes to sleep or starts doinganother work, and the system needs to grasp how far the arousal recoveryof consciousness required for driving recovery is progressing. That is,in a conventional driver monitoring system, a main detection meansdetects a decrease in consciousness such as drowsiness. However, in thefuture, the driver will be completely uninvolved in the driving andsteering. Therefore, the system has no means for directly observing anintervention level of the driver from steering stability of a steeringdevice and the like, and needs to observe a consciousness recoverytransition required for driving from a state where an accurateconsciousness level of the driver is unknown, grasp an accurate internalarousal state of the driver, and proceed in intervention in the manualdriving of steering from the automatic driving.

Therefore, the vehicle interior information detection unit 142 mainlyhas two major roles. The first role is passive monitoring of thedriver's state during the automatic driving. The second role is todetect the driver's periphery recognition, perception, judgment, and anoperation ability of the steering device up to the level at which themanual driving is possible from when the recovery request is issued fromthe system to when the vehicle approaches a section of driving undercaution. As control, a failure self-diagnosis of the entire vehicle maybe further performed, and in a case where the function of the automaticdriving is deteriorated due to partial malfunction of the automaticdriving, the driver may be similarly prompted to recover to the manualdriving early. The passive monitoring here refers to a type of detectionmeans that does not require a conscious response reaction from thedriver, and does not exclude devices that detect a response signal bytransmitting physical radio waves, light, or the like from the device.That is, the passive monitoring refers to monitoring of the driver'sunconscious state, such as during a nap, and classification that is notthe driver's cognitive response is a passive system. The passivemonitoring does not exclude active response devices that analyze andevaluate reflected or diffused signals obtained by emitting radio waves,infrared rays, or the like. Meanwhile, devices requesting the driver togive a conscious response requesting a response reaction are activesystems.

The environment in the vehicle to be detected includes, for example,temperature, humidity, brightness, odor, and the like. The vehicleinterior information detection unit 142 supplies data indicating resultsof the detection processing to the situation recognition unit 153 of thesituation analysis unit 133 and the operation control unit 135. Notethat, in the case where it is revealed that the driver cannot achievethe manual driving within an appropriate deadline after the drivingrecovery instruction to the driver is issued from the system, and it isdetermined that the takeover will not be in time even if decelerationcontrol is performed in self-operation to give a time, an instruction isgiven to the emergency avoidance unit 171 and the like of the system,and deceleration, evacuation, and stop procedures are started forevacuating the vehicle. That is, even in a situation where the takeovercannot be in time as an initial state, it is possible to earn time toreach a takeover limit by starting the deceleration of the vehicleearly.

The vehicle state detection unit 143 performs processing of detectingthe state of the user's own car on the basis of data or signals fromeach unit of the mobile device 100. The state of the user's own car tobe detected includes, for example, speed, acceleration, steering angle,presence or absence of abnormality, content of abnormality, state ofdriving operation, position and tilt of a power seat, a state of doorlock, states of other in-vehicle devices, and the like. The vehiclestate detection unit 143 supplies data indicating results of thedetection processing to the situation recognition unit 153 of thesituation analysis unit 133, the emergency avoidance unit 171 of theoperation control unit 135, and the like.

The self-position estimation unit 132 performs processing of estimatingthe position, posture, and the like of the user's own car on the basisof the data and signals from the units of the mobile device 100, such asthe vehicle exterior information detection unit 141 and the situationrecognition unit 153 of the situation analysis unit 133. Furthermore,the self-position estimation unit 132 generates a local map (hereinafterreferred to as self-position estimation map) to be used for estimatingthe self-position, as needed.

The self-position estimation map is a high-precision map using atechnology such as simultaneous localization and mapping (SLAM), or thelike. The self-position estimation unit 132 supplies data indicating aresult of the estimation processing to the map analysis unit 151, thetraffic rule recognition unit 152, and the situation recognition unit153 of the situation analysis unit 133, and the like. Furthermore, theself-position estimation unit 132 causes the storage unit 111 to storethe self-position estimation map.

The situation analysis unit 133 performs processing of analyzing thesituation of the user's own car and its surroundings. The situationanalysis unit 133 includes the map analysis unit 151, the traffic rulerecognition unit 152, the situation recognition unit 153, a situationprediction unit 154, and a safety determination unit (learningprocessing unit) 155.

The map analysis unit 151 performs processing of analyzing various mapsstored in the storage unit 111, using the data or signals from the unitsof the mobile device 100, such as the self-position estimation unit 132and the vehicle exterior information detection unit 141, as needed, andbuilds a map including information necessary for automatic drivingprocessing. The map analysis unit 151 supplies the built map to thetraffic rule recognition unit 152, the situation recognition unit 153,the situation prediction unit 154, and a route planning unit 161, anaction planning unit 162, and an operation planning unit 163 of theplanning unit 134, and the like.

The traffic rule recognition unit 152 performs processing of recognizinga traffic rule around the user's own car on the basis of the data orsignals from the units of the mobile device 100, such as theself-position estimation unit 132, the vehicle exterior informationdetection unit 141, and the map analysis unit 151. By the recognitionprocessing, for example, the position and state of signals around theuser's own car, the content of traffic regulation around the user's owncar, a travelable lane, and the like are recognized. The traffic rulerecognition unit 152 supplies data indicating a result of therecognition processing to the situation prediction unit 154 and thelike.

The situation recognition unit 153 performs processing of recognizingthe situation regarding the user's own car on the basis of the data orsignals from the units of the mobile device 100, such as theself-position estimation unit 132, the vehicle exterior informationdetection unit 141, the vehicle interior information detection unit 142,the vehicle state detection unit 143, and the map analysis unit 151. Forexample, the situation recognition unit 153 performs processing ofrecognizing the situation of the user's own car, the situation aroundthe user's own car, the situation of the driver of the user's own car,and the like. Furthermore, the situation recognition unit 153 generatesa local map (hereinafter referred to as situation recognition map) usedfor recognizing the situation around the user's own car, as needed. Thesituation recognition map is, for example, an occupancy grid map.

The situation of the user's own car to be recognized is, for example,the position, posture, and motion of the user's own car (for example,speed, acceleration, moving direction, and the like), and a cargo loadcapacity and movement of the center of gravity of the vehicle bodyaccompanying cargo loading, a tire pressure, a braking distance movementaccompanying wear of a braking pad, allowable maximum decelerationbraking to prevent cargo movement caused by load braking, and acentrifugal relaxation limit speed at the time of traveling on a curvewith a liquid load, which are specific to the vehicle and determiningmotion characteristics of the user's own car. Moreover, the recoverystart timing required for control is different depending on theconditions specific to the loading cargo, the characteristics specificto the vehicle itself, the load, and the like even if the roadenvironment such as a friction coefficient of a road surface, a roadcurve, or a slope is exactly the same. Therefore, such variousconditions need to be collected and learned, and reflected in theoptimal timing for performing control. Simply observing and monitoringthe presence or absence and content of abnormality of the user's ownvehicle, for example, is not sufficient in determining the controltiming according to the type of the vehicle and the load. To secure acertain level of safety in the transportation industry, or the like,according to unique characteristics of the load, parameters fordetermining addition of time for desired recovery may be set as a fixedvalue in advance, and it is not always necessary to uniformly set allnotification timing determination conditions by self-accumulationlearning.

The situation around the user's own car to be recognized include, forexample, types and positions of surrounding stationary objects, types ofsurrounding moving objects, positions and motions (for example, speed,acceleration, moving direction, and the like), configurations ofsurrounding roads and conditions of road surfaces, as well assurrounding weather, temperature, humidity, brightness, and the like.The state of the driver to be recognized includes, for example, physicalcondition, arousal level, concentration level, fatigue level,line-of-sight motion, traveling operation, and the like. To cause thevehicle to safely travel, a control start point requiring measuresgreatly differs depending on a loading capacity mounted in a statespecific to the vehicle, a chassis fixed state of a mounting unit, adecentered state of the center of gravity, a maximum decelerableacceleration value, a maximum loadable centrifugal force, a recoveryresponse delay amount according to the state of the driver, and thelike.

The situation recognition unit 153 supplies data indicating a result ofthe recognition processing (including the situation recognition map, asneeded) to the self-position estimation unit 132, the situationprediction unit 154, and the like. Furthermore, the situationrecognition unit 153 causes the storage unit 111 to store the situationrecognition map.

The situation prediction unit 154 performs processing of predicting thesituation regarding the user's own car on the basis of the data orsignals from the units of the mobile device 100, such as the mapanalysis unit 151, the traffic rule recognition unit 152, and thesituation recognition unit 153. For example, the situation predictionunit 154 performs processing of predicting the situation of the user'sown car, the situation around the user's own car, the situation of thedriver, and the like.

The situation of the user's own car to be predicted includes, forexample, a behavior of the user's own car, occurrence of abnormality, atravelable distance, and the like. The situation around the user's owncar to be predicted includes, for example, a behavior of a moving bodyaround the user's own car, a change in a signal state, a change in theenvironment such as weather, and freezing of a road surface, snow,wetness, sand accumulation, and the like that occur as a result of theweather. The situation of the driver to be predicted includes, forexample, a behavior of the driver and physical conditions of the driver,which may affect driving and steering characteristics.

The situation prediction unit 154 supplies data indicating a result ofthe prediction processing together with the data from the traffic rulerecognition unit 152 and the situation recognition unit 153 to the routeplanning unit 161, the action planning unit 162, the operation planningunit 163 of the planning unit 134, and the like.

The safety determination unit (learning processing unit) 155 has afunction as a learning processing unit that learns optimal recoverytiming according to a recovery action pattern of the driver, the vehiclecharacteristics, and the like, and provides learned information to thesituation recognition unit 153 and the like. As a result, for example,it is possible to present to the driver statistically determined optimumtiming required for the driver to normally recover from the automaticdriving to the manual driving at a predetermined ratio or more.

The route planning unit 161 plans a route to a destination on the basisof the data or signals from the units of the mobile device 100, such asthe map analysis unit 151 and the situation prediction unit 154. Forexample, the route planning unit 161 sets a route to a destinationspecified from a current position on the basis of the global map.Furthermore, for example, the route planning unit 161 appropriatelychanges the route on the basis of situations of congestion, accidents,traffic regulations, construction, and the like, the physical conditionsof the driver, and the like. The route planning unit 161 supplies dataindicating the planned route to the action planning unit 162 and thelike.

The action planning unit 162 plans an action of the user's own car forsafely traveling in the route planned by the route planning unit 161within a planned time on the basis of the data or signals from the unitsof the mobile device 100 such as the map analysis unit 151 and thesituation prediction unit 154. For example, the action planning unit 162makes a plan of starting, stopping, traveling directions (for example,forward, backward, turning left, turning right, turning, and the like),driving lane, traveling speed, passing, and the like. The actionplanning unit 162 supplies data indicating the planned action of theuser's own car to the operation planning unit 163 and the like.

The operation planning unit 163 plans an operation of the user's own carfor implementing the action planned by the action planning unit 162 onthe basis of the data or signals from the units of the mobile device100, such as the map analysis unit 151 and the situation prediction unit154. For example, the operation planning unit 163 plans acceleration,deceleration, a traveling track, and the like. The operation planningunit 163 supplies data indicating the planned motion of the user's owncar to an acceleration and deceleration control unit 172 and a directioncontrol unit 173 of the operation control unit 135, and the like.

The operation control unit 135 controls the operation of the user's owncar. The operation control unit 135 includes the emergency avoidanceunit 171, the acceleration and deceleration control unit 172, and thedirection control unit 173.

The emergency avoidance unit 171 performs processing of detecting anemergency situation such as collision, contact, entry into a dangerzone, driver's abnormality, vehicle's abnormality, and the like on thebasis of the detection results of the vehicle exterior informationdetection unit 141, the vehicle interior information detection unit 142,and the vehicle state detection unit 143. In the case where theemergency avoidance unit 171 detects occurrence of the emergencysituation, the emergency avoidance unit 171 plans the operation of theuser's own car for avoiding the emergency situation, such as sudden stopor sharp turn. The emergency avoidance unit 171 supplies data indicatingthe planned operation of the user's own car to the acceleration anddeceleration control unit 172, the direction control unit 173, and thelike.

The acceleration and deceleration control unit 172 performs accelerationand deceleration for implementing the operation of the user's own carplanned by the operation planning unit 163 or the emergency avoidanceunit 171. For example, the acceleration and deceleration control unit172 calculates a control target value of a drive force generation deviceor a braking device for implementing the planned acceleration,deceleration, or sudden stop, and supplies a control command indicatingthe calculated control target value to the drive system control unit107. Note that, there are two main cases where an emergency situationoccurs. That is, there are a case where an unexpected accident hasoccurred due to a sudden reason during the automatic driving on a roadon a traveling route, which is originally supposed to be safe accordingto the local dynamic map or the like acquired from an infrastructure andan emergency recovery cannot be in time, and a case where the driver hasa difficulty in accurately recovering to the manual driving from theautomatic driving.

The direction control unit 173 controls a direction for implementing theoperation of the user's own car planned by the operation planning unit163 or the emergency avoidance unit 171. For example, the directioncontrol unit 173 calculates a control target value of a steeringmechanism for implementing the traveling track or sharp turn planned bythe operation planning unit 163 or the emergency avoidance unit 171, andsupplies a control command indicating the calculated control targetvalue to the drive system control unit 107.

[3. Mode Switching Sequence from Automatic Driving Mode to ManualDriving Mode]

Next, a takeover sequence from the automatic driving mode to the manualdriving mode will be described.

FIG. 11 schematically illustrates an example of a mode switchingsequence from the automatic driving mode to the manual driving mode inthe automatic driving control unit 112.

In step S1, the driver is in a state of being completely detached fromthe driving and steering. In this state, for example, the driver canexecute a secondary task such as taking a nap, watching a video,concentrating on a game, and working with a visual tool such as a tabletor a smartphone. The work using the visual tool such as a tablet or asmart phone may be performed, for example, in a state where the driver'sseat is displaced or in a seat different from the driver's seat.

When the vehicle approaches a section requiring manual driving recoveryon the route, it is assumed that the time until the driver recoversgreatly varies depending on the operation content at that time. With thenotification just before the approach to the event, the time isinsufficient to recover. In a case where the notification is made tooearly with respect to the approach of the event with a margin, the timeto the timing actually required for recovery may be too long, dependingon the state of the driver. As a result, if the situation where thenotification is not performed at appropriate timing repeatedly occurs,the driver loses the reliability for the notification timing of thesystem, and the driver's consciousness for the notification decreases,and the driver's accurate treatment is neglected, accordingly. As aresult, the risk of failing in takeover is increased, and at the sametime, it becomes a factor to hinder comfort execution of the secondarytask. Therefore, to enable the driver to start accurate driving recoveryto the notification, the system needs to optimize the notificationtiming.

Step S2 is the timing of the manual driving recovery requestnotification described above with reference to FIG. 2 and FIGS. 5 to 7.Notification of the driving recovery is issued to the driver usingdynamic puptics such as vibration or a visual or auditory manner. Theautomatic driving control unit 112 monitors a steady state of thedriver, for example, grasps the timing to issue the notification, andissues the notification at appropriate timing. That is, the systempassively and constantly monitors the driver's secondary task executionstate during the former passive monitoring period and can calculateoptimal timing of the notification. It is desirable to continuously andconstantly perform the passive monitoring in the period of step S1 andto calculate the recovery timing and issue the recovery notificationaccording to recovery characteristics unique to the driver.

That is, it is desirable to learn the optimal recovery timing accordingto the recovery action pattern of the driver, the vehiclecharacteristics, and the like, and to present, to the driver, thestatistically obtained optimal timing, which is required for the driverto normally recover from the automatic driving to the manual driving ata predetermined rate or higher. In this case, in a case where the driverdoes not responded to the notification for a certain period of time, awarning by sounding an alarm or the like is given.

In step S3, whether or not the driver has been seated and recovered isconfirmed. In step S4, an internal arousal state of the driver isconfirmed by analyzing a face or an eyeball behavior such as saccade. Instep S5, stability of an actual steering situation of the driver ismonitored. Then, in step S6, the takeover from the automatic driving tothe manual driving is completed.

FIG. 12 illustrates a further detailed example of the takeover sequencefrom the automatic driving to the manual driving. In step S11,prediction of a recovery point is visually presented to theabove-described information terminal 50, or a visual tool such as atablet or a smartphone. Note that, it is not necessary to limit thedisplay to the visual tools, and a display form such as a centerinformation display of the vehicle is desirable, which enters thedriver's field of vision during execution of a secondary task. Althoughdetails will be described below, the forward schedule and approachinformation are presented, and the recovery point is displayed so as toapproach the user's own car over time.

In step S12, presentation content of the forward schedule and approachinformation is changed as appropriate by updating a so-called localdynamic map (LDM) in which road environment information, for example,travel map information of roads on which the vehicle travels is updatedwith high density and on a constant basis. Furthermore, in this stepS12, the state of the driver is regularly monitored.

In step S13, a time to a recovery point (a recovery point from theautomatic driving to the manual driving) within a fixed time from thecurrent point is displayed in a confirmable manner on the informationterminal 50 worn by the driver. For example, examples are described withreference to FIGS. 6 and 7. Moreover, when the time to the manualdriving start position is within a predetermined time, the notificationdisplay requesting recovery to the manual driving described withreference to FIG. 5 is made. The timing of issuing the notification isadjusted such that the driver can recover in time by executing thenotification early according to a detection result of the regularmonitoring in the previous stage, that is, according to the depth ofdetachment from the driving by nap or a secondary task.

In step S14, if the driver does not respond to the notification, analarm for waking up is sounded. In step S15, if the driver is notseated, the driver is visually or auditorily notified to recover. Instep S16, in a case where there is a delay in returning to seating, awarning alarm is sounded. In step S17, for example, the driver'spointing and calling for checking forward is monitored as recoverystart.

In step S18, the sitting posture is confirmed. In step S19, the internalarousal level of the driver is determined by perceptual reflexes such assaccade or fixation in order to detect recovery of an internal brainperception state of the driver, using a means such as a detailedline-of-sight analysis, for example. In step S20, the driver issequentially entrusted with driving steering authority, the steering isentrusted while observing a response steering situation of an actualsteering reaction, and the steering stability is monitored.

As a result of monitoring and observing the stability, if the driver'sproactive driving steering recovery is not detected to the expectedextent, it means that there is a risk that a dreaming state is stillextended. Therefore, in step S21, when it is expected that normalrecovery is impossible, it is determined that the emergency takeover hasfailed, and a deceleration slowdown evacuation sequence is started. Notethat, although detailed description in each step is omitted, thenotification is once made to the driver, and in a case where therecovery transition of the driver requires time than assumed recoverytime and exhaustion risk of recovery time is estimated in each step ofcontinuously monitoring the recovery transition, the deceleration may bestarted in stages. The actual deceleration amount is determined on thebasis of optimization decision of influence in consideration ofinfluence caused by the deceleration of the user's own vehicle.

The temporal transition for recovery varies by various factors such asage, experience, fatigue level of the driver, and is thus determined byrecovery timing according to the individual. In the case where thedriver is requested to recover from the automatic driving to the manualdriving, it takes at least a certain time to almost certainly recover tothe manual driving. The most desirable configuration is to notify theoptimal timing according to vehicle characteristics, roadcharacteristics, and personal driving recovery characteristics. Bydisplaying the approach situation of a takeover point in conjunctionwith timing, the driver can feel relieved and gain convenience of beingable to use the secondary task according to an allowable situation, andat the same time, since a tense state and a relaxed state appearsalternately, the user can appropriately recover to the attention in eachnecessary section, which ergonomically leads to a suitable use. In otherwords, since the driver is free from unnecessary continuous tension, itis possible to reduce induction of drowsiness and distraction in thedriver by allowing traveling under moderate caution from the automaticdriving that continuously requires traveling under constant high cautionfor a long time.

The greatest effect of these procedures is to balance the secondarytasks in the user's state of tension and relaxation. In the conventionalnotification method of performing uniform notification based onposition, there is no choice but to predict the actual takeover timingwith the driver's sense. The driver has to keep the tension in mindwhile keeping an eye on when a request to take over is given, getsnervous, and as a result, there is a risk of becoming drowsy,distracted, and the like. In contrast, in the takeover timingnotification to which the situation by the disclosed method is applied,there is an effect that the driver's continuous tension/attention statecan be appropriately reduced according to a section. Note that, tomaximize the effect, it is favorable to appropriately generate a dummyrecovery request event in the middle of itinerary even in a section notrequiring the driver's intervention over a long distance. Recovery levelevaluation of the driver and aptitude reaction evaluation of the drivermay be performed according to the recovery level of the driver to thedummy event, and recovery level evaluation values that occurs each timemay be further recorded and stored. It is favorable to perform theactive response evaluation of the driver using the dummy event whenespecially the driver passes through a section with a high eventoccurrence rate where it is difficult for the driver to handle with deepsleep. Furthermore, the evaluation may be executed for accuratelymonitoring the state of the driver when passing through a section notinterfering with a secondary task that allows a relatively short-termrecovery of several tens of seconds, or when leaving the driver's seatand working in the luggage compartment or the like. Alternatively, theevaluation may be executed to complementarily observe the driver's statewhen the system loses the driver's state at the time of detecting apassive part signal.

[4. Operation Sequence Example of Automatic Driving]

Next, an example of an operation sequence of the automatic driving willbe described with reference to the flowchart illustrated in FIG. 13.

The flowchart illustrated in FIG. 13 is a flowchart for describing theoperation sequence of the automatic driving executed by the mobiledevice 100.

First, in step S21, driver authentication is performed. This driverauthentication is performed using knowledge authentication using apassword, a PIN, or the like, biometric authentication using the face, afingerprint, an iris of a pupil, a voice print, or the like, or theknowledge authentication and the biometric authentication together. Byperforming the driver authentication in this way, information fordetermining the notification timing can be selected and accumulated inassociation with each driver even in a case where a plurality of driversdrives the same vehicle.

Next, in step S22, the driver operates the input unit 101 to set adestination. In this case, the driver's input operation is performed onthe basis of display on an instrument panel.

Note that the present embodiment has been described assuming the casewhere the driver gets in the vehicle and sets the itinerary. However,the driver may remotely make a reservation in advance using a smartphonebefore getting in the vehicle or using a personal computer or adedicated terminal before leaving home or a distribution center, forexample. Moreover, the system of the vehicle may perform preplanningalong a schedule assumed by the driver according to a schedule table,update and acquire the LDM information of the road environment, that is,the so-called local dynamic map (LDM) in which road traveling mapinformation of the vehicle is updated with high density and on aconstant basis, and display an advice of actual traveling at the time ofor before getting in the vehicle, like a concierge, for example.

Next, in step S23, display of a traveling section on a traveling routeis started. This traveling section is displayed on the instrument paneland is also displayed with a work window on a tablet or the like onwhich the driver performs a secondary task, for example. As a result,the driver working on the work window can easily recognize a driverintervention required section and an automatically drivable section onthe traveling route on a predicted arrival time axis from the currentpoint.

In the traveling section display, a forward schedule and approachinformation to each point are presented. In the traveling sectiondisplay, the driver intervention required section and the automaticallydrivable section on the traveling route are displayed on the predictedarrival time axis from the current point. Then, the driver interventionrequired section includes a manual driving section, a takeover sectionfrom the automatic driving to the manual driving, and a cautionedtraveling section from the automatic driving. Details of the travelingsection display will be described below.

Next, in step S24, acquisition of LDM update information is started.With the acquisition of LDM update information, content of the travelingsection display can be changed to the latest state. Next, in step S25,traveling is started. Next, in step S26, the traveling section displayis updated on the basis of the position information of the user's owncar and the acquired LDM update information. For example, the sectioninformation described with reference to FIGS. 6 and 7 is displayed onthe information terminal 50.

Next, in step S27, the driver's state is monitored. Next, in step S28,event change handling processing is performed. The event change handlingprocessing includes mode switching processing for responding to a casewhere a switching point between the automatic driving mode and themanual driving mode existing on the traveling route or the cautionedtraveling section is approaching, event occurrence processing forresponding to a case where the mode switching point or the driverintervention required section of the cautioned traveling section newlyoccurs on the traveling route, and the like. Hereinafter, the processesof steps S26 to S28 are appropriately repeated. In the case where thereis an event change, it can be dealt with in step S28, and details willbe described below.

“Details of Traveling Section Display” FIG. 14 illustrates an example ofthe traveling route determined as the destination is determined by thedriver. At least an automatic driving section Sa and a manual drivingsection Sb are set for the traveling route. Moreover, the example inFIG. 14 illustrates a configuration in which a takeover section Sc fromthe automatic driving to the manual driving and a cautioned travelingsection Sd from the automatic driving are set in a case of assuming aroute traveling counterclockwise. The traveling section information isinformation of takeover points presented to the traveling vehicleaccording to the operation of the user's own vehicle on the basis of theinformation recorded in the local dynamic map (LDM).

Here, the takeover section Sc exists immediately before the manualdriving section Sb, and the driver needs to be in a recoverable state tothe manual driving. Furthermore, the cautioned traveling section Sd is asection in which the vehicle can travel while decelerating while keepingthe automatic driving under caution of the driver who is in therecoverable state to the manual driving. The display of the recoverynotification point is not necessarily appropriate in the presentation ofa relatively wide range of map information, and selective display may beperformed. The reason is that the notification timing varies due to thesecondary task that can be taken in each driving.

In the illustrated example, the automatic driving section Sa isillustrated in green, the manual driving section Sb is illustrated inred, and the takeover section Sc and the cautioned traveling section Sdare illustrated in yellow. Note that, for the sake of convenience, eachcolor is represented by a different pattern.

As described with reference to FIGS. 6 and 7, the section information isdisplayed on the display or the tablet of the information terminal 50 orthe automobile 10. As described above with reference to FIGS. 6 and 7,each section on the traveling route is displayed on the display deviceof the information terminal 50 and the like with the predicted arrivaltime axis from the current point. The data processing unit 11 of theautomobile 10 illustrated in FIG. 1, specifically, the automatic drivingcontrol unit 112 of the mobile device 100 illustrated in FIG. 8calculates an estimated arrival time of each section for displaying thetraveling section on the traveling route on the basis of traveling routeinformation and traffic information. This calculated data is transmittedto the information terminal 50 via the communication unit. Theinformation terminal 50 performs the data display described withreference to FIGS. 6 and 7, for example, on the basis of the receiveddata.

Details of the data displayed on the information terminal 50 will bedescribed with reference to FIG. 15. FIG. 15 illustrates a linearlyconverted traveling route of the automobile.

FIG. 15(a) illustrates each section on the traveling route on a constantscale with a moving distance axis from the current point. FIG. 15(b)illustrates an average vehicle flow velocity v(t) at each point. Thedata in FIG. 15(b) can be acquired by the automobile 10 by receiving theLDM distributed by a local dynamic map (LDM) distribution server. FIG.15(c) is a diagram in which each section represented by the movingdistance axis is converted into a time axis using the velocity v(t). Thedata processing unit 11 of the automobile 10 illustrated in FIG. 1generates data (c) from data (a) and (b) illustrated in FIG. 15. As aresult, each section on the traveling route is represented by thepredicted arrival time axis from the current point. That is, eachsection on the traveling route can be represented by the time axisobtained by dividing the physical distance of the traveling route by theaverage velocity of each corresponding section. The data processing unit11 of the automobile 10 transmits the generated data to the informationterminal 50 via the communication unit. The information terminal 50performs the data display described with reference to FIGS. 6 and 7, forexample, on the basis of the received data.

The display data of the information terminal 50 described with referenceto FIGS. 6 and 7 is data obtained by converting the data illustrated inFIG. 15(c) into a pie chart and making the data correspond to the timeaxis (360 degrees=10 minutes) of the information terminal 50.

FIG. 16 illustrates a correspondence example between the data in FIG.15(c) and the display data of the information terminal 50. Asillustrated in FIG. 16, the display data of the information terminal 50illustrated in FIG. 16 is data obtained by converting the data (c) of 0to 10 (minutes) on the time axis into a pie chart and making the datacorrespond to the time axis (360 degrees=10 minutes) of the informationterminal 50.

The driver 20 can grasp the remaining time to the manual driving sectionby seeing the display data of the information terminal 50. Since theinformation terminal 50 is a wearable terminal worn on the arm of thedriver 20, even when the driver 20 is away from the driver's seat andworking, the remaining time to the manual driving section can always beconfirmed. Furthermore, as described above, the information terminal 50has an alarm output function and a vibration function, and for example,when the remaining time to the manual driving section Sb (=red display)reaches a predefined time (for example, 1 minute), the informationterminal 50 outputs an alarm and activates vibration to warn the driver20. An optimum execution time of the alarm and notification will bedescribed below. First, a case of using a fixed value will be described.Note that, at the time of warning output, the display of the informationterminal 50 may be switched to the notification screen for switching theautomatic driving to the manual driving described above with referenceto FIG. 5. However, when switching is performed, the remaining time tothe manual driving section cannot be checked. Therefore, it is favorableto perform processing such as semi-transparent display of the switchingnotification or displaying the remaining time in a part of the displayarea. Note that the alarm, vibration, display switching processing, andtiming can be changed or stopped according to user settings.

The display unit of the information terminal 50 may be configured toperform the display illustrated in FIG. 17, in addition to the displayillustrated in FIG. 16, for example. The display example of theinformation terminal 50 illustrated in FIG. 17 is a displayconfiguration in which the road section information of 0 to 10 minutesfrom the present time (the automatic driving section, the manual drivingsection, and the like) is displayed in an inner circumference of thedisplay unit of the information terminal 50, and the road sectioninformation of 0 to 60 minutes from the present time or from a pointthereafter (the automatic driving section, the manual driving section,and the like) is displayed in an outer circumference. By the display,the driver 20 can visually and intuitively recognize the road sectioninformation of 0 to 60 minutes from the present time in addition to theroad section information of 0 to 10 minutes from the present time.

Moreover, the configuration of performing the display in FIG. 18 may beadopted. The display example of the information terminal 50 illustratedin FIG. 18 is a display configuration in which the road sectioninformation of 0 to 10 minutes from the present time (the automaticdriving section, the manual driving section, and the like) is displayedin an innermost circumference of the display unit of the informationterminal 50, the road section information of 0 to 60 minutes from thepresent time (the automatic driving section, the manual driving section,and the like) is displayed in a second inner circumference outside theinnermost circumference, and the road section information of 0 to 240minutes from the present time (the automatic driving section, the manualdriving section, and the like) is displayed in an outer circumferenceoutside the second inner circumference. By the display, the driver 20can grasp the road section information of 0 to 60 minutes and 0 to 240minutes from the present time in addition to the road sectioninformation of 0 to 10 minutes from the present time. By the displayprocessing using these plural time axes, the driver can know the timingto recover in a bird's eye view manner, and an action plan becomes easy.Furthermore, the information understood by the brain becomesmultidimensional, and it becomes easy to understand multidimensionalinformation such as arrival time of each section, importance ofintervention, intervening period, and intermediate distribution ofintervening points.

[5. Sequence of Processing Executed by Mobile Device and InformationProcessing Device (Information Terminal) of Present Disclosure]

Next, a sequence of processing executed by the mobile device and theinformation processing device (information terminal) of the presentdisclosure will be described.

The flowchart illustrated in FIG. 19 is a flowchart for describing anexample of a sequence of processing executed when an automobile as themobile device of the present disclosure switches the automatic drivingmode to the manual driving mode.

In step S41, the mobile device observes the occurrence event of therequest for switching the automatic driving mode to the manual drivingmode.

This observation information is information acquired from the localdynamic map (LDM) that holds the road environment information and thelike distributed by the local dynamic map (LDM) distribution server tothe automobile. That is, the observation information is acquired on thebasis of the local dynamic map (LDM) information in which the travel mapinformation of roads on which the vehicle travels is updated with highdensity and on a constant basis.

As a specific example, for example, in the case where a schedule tostart entry to a general road from a highway is acquired on the basis ofthe LDM, it is determined that occurrence of an event of requestingswitching from the automatic driving mode to the manual driving mode.

Next, in step S42, the observation value is acquired. The observationvalue acquisition processing is performed, for example, in the driverinformation acquisition unit 12 and the environment informationacquisition unit 13 of the automobile 10 illustrated in FIG. 1 andfurther using the biosensor (see FIG. 10) of the information terminal50, and the like. Note that the information terminal 50 of theautomobile 10 executes communication constantly or intermittently totransmit or receive information required by both devices.

The driver information acquisition unit 12 of the automobile 10 includesa camera and various sensors, and acquires the driver information, suchas information for determining the arousal level of the driver, forexample. The information is, for example, a line-of-sight direction, aneyeball behavior, and a pupil diameter acquired from an image includingan eyeball area, and a facial expression acquired from an imageincluding a face area. The driver information acquisition unit 12further acquires the operation information of the operation units(steering wheel, accelerator, brake, and the like) of the driver.

The environment information acquisition unit 13 acquires, for example,an image by an imaging unit installed in the mobile device, depthinformation, three-dimensional structure information, topographicalinformation by sensors such as an LiDAR installed on a moving body,position information by a GPS, traffic light conditions, signinformation, information from a communication device installed on aninfrastructure such as a road, and the like.

Next, in step S43, the manual driving recoverable time is calculated onthe basis of the acquired observation value.

This processing is processing executed by the data processing unit 11illustrated in FIG. 1.

The data processing unit 11 illustrated in FIG. 1 receives, for example,the driver information acquired by the driver information acquisitionunit 12, the environment information acquired by the environmentinformation acquisition unit 13, and the position and biometricinformation of the driver acquired by the information terminal 50 asinputs. Moreover, the data processing unit 11 estimates the time(=manual driving recoverable time) required by safe manual drivingrecovery on the basis of the current driver information and environmentinformation, using the learning result corresponding to the driver inthe data processing unit 11 (the safety determination unit (learningprocessing unit) 155 in FIG. 8) or the learning result corresponding tothe driver acquired from the external server 30.

Note that, in the processing of estimating the time (=manual drivingrecoverable time) required by the safe manual driving recovery, theposition of the driver acquired by the information terminal 50 is alsotaken into consideration. That is, in the case where the driver 20 is ata position distant from the driver's seat, for example, in the luggagecompartment 21, the time is calculated in consideration of the time toreturn to the driver's seat.

The learning result corresponding to the driver in the data processingunit 11 of the automobile 10 illustrated in FIG. 1 (the safetydetermination unit (learning processing unit) 155 illustrated in FIG.8), or the learning result corresponding to the driver acquired from theexternal server 30 is learning data based on past driving data of thedriver and is stored in the storage unit of the automobile 10 or thestorage unit of the server 30. As a learning method to be applied, forexample, Support Vector Machine, Boosting, Convolutional Neural Network,Deep Belief Network, Long Short-Term Memory, or the like can beconsidered.

The data processing unit 11 of the automobile 10 illustrated in FIG. 1(safety determination unit (learning processing unit) 155 illustrated inFIG. 8) estimates the time (=manual driving recoverable time) requiredby the safe manual driving recovery, using the learning data and theinput data (driver information and environment information).

Note that the learning device used in the processing of estimating themanual driving recoverable time can be set for each driver or set toinclude the type of the secondary information during the automaticdriving to the observation information.

In this case, the processing (manual driving recoverable time estimationprocessing) using the personal identification information of the driverwho is currently driving and the information of the type of thesecondary task being currently executed as the observation informationis performed.

FIG. 20(a) illustrates an example of distribution of a plurality ofpieces of relationship information (observation plots) between theobservable evaluation value corresponding to an observation value andthe recovery delay time (=manual driving recoverable time). This examplecorresponds to a type of a certain secondary task of a certain driver.To calculate the recovery delay time from the plurality of pieces ofrelationship information (observation plots), the relationshipinformation (observation plot) in an area (illustrated by thebroken-line rectangular frame) having a certain width in an evaluationvalue direction corresponding to the acquired observation value isextracted. A dotted line c in the figure represents a boundary line ofwhen the recovery delay time at which the recovery ratio is 0.95 in FIG.20(b) described below is observed with different observation values ofthe driver.

By issuing the recovery notification from the automatic driving to themanual driving or an alarm to the driver for a longer time, that is, inan earlier time, than the dotted line c, the driver's successfulrecovery from the automatic driving to the manual driving is secured atthe ratio of 0.95 or higher. Note that a target value (request forrecovery ratio) for allowing the driver to normally recover from theautomatic driving to the manual driving for each corresponding sectionis determined by the roadside from the necessity of infrastructure, forexample, and is provided to the individual vehicle passing through thesection

Note that, in a case where the vehicle does not interfere withsurroundings even if the vehicle stops on the road, the vehicle is onlyrequired to be stopped, or the vehicle is only required to bedecelerated to the speed handleable by the system. Normally, stopping avehicle on a traveling road is not always desirable, and therefore, ahigh recovery ratio is desirable as a default setting. In particular, ina specific route such as metropolitan expressway, an extremely highrecovery ratio may be required even if the infrastructure does notprovide update information.

FIG. 20(b) illustrates a relationship between the recovery delay timeand the recovery ratio obtained from the plurality of pieces ofextracted relationship information (observation plots). Here, a curve aillustrates an independent success ratio at each recovery delay time,and a curve b illustrates a cumulative success ratio at each recoverydelay time. In this case, a recovery delay time t1 is calculated suchthat the success ratio becomes a predetermined ratio, that is, thesuccess ratio becomes 0.95 in the illustrated example, on the basis ofthe curve b.

This calculation processing is executed in the data processing unit 11of the automobile 10 in FIG. 1 (safety determination unit (learningprocessing unit) 155 in FIG. 8). For example, the calculation processingis performed using the distribution information of the plurality ofpieces of relationship information (observation plots) between theobservable evaluation value and the recovery delay time stored in andacquired from the storage unit in the past.

FIG. 21 is a graph for describing the manual driving recoverable timeaccording to a type of processing (secondary task) executed by thedriver in the automatic driving mode when the driver is detached fromthe driving and steering operation.

Each distribution profile corresponds to the curve a illustrated in FIG.20(b), which is predicted on the basis of the observed value, that is,the driver state. That is, to complete the takeover from the automaticdriving to the manual driving at the takeover point with a necessaryrecovery ratio, whether or not a state actually reaches a necessarystate required for recovery at each recovery stage is monitored untilthe takeover is completed on the basis of the time t1 when the profile(the recovery ratio profile in FIG. 20(b)) becomes a desired value byreference to the past characteristics required for the driver torecovery, from observation values capable of evaluating the arousallevel of the driver detected at each stage.

For example, the initial curve in the case of taking a nap hascumulative average distribution in the case of estimating a sleep levelfrom observation information such as breathing and pulse waves that arepassively monitored during the nap period in the automatic driving, andviewing recovery delay characteristics of the driver after issuing awakeup alarm. Each halfway distribution is determined according to thedriver's state observed after the driver wakes up and in a subsequentmovement recovery procedure. “6. In the case of taking a nap”illustrated in the drawing is observed and the right timing in time forthe wakeup alarm is determined, and a halfway process thereafter showsthe recovery time distribution in a recovery budget predicted from anobservable driver state evaluation value at a predicted intermediatepoint.

Observation as to not violating a remaining takeover time limit, whichgradually decreases until the takeover, is continued halfway, and in thecase where there is a violation risk, the vehicle is decelerated, and atime delay is generated, for example. Note that, for example, regardingdistribution of recovery starting from “4. Non-driving posture irregularrotation seating” without the steps of “6. In the case of taking a nap”and “5. Seated”, the process of recovery starts from initial situationrecognition grasping. Therefore, in the case of starting from thesituation recognition in the “4. Non-driving posture irregular rotationseating” posture from the beginning, the time to recognize the situationis long. Whereas in the state of “4. Non-driving posture irregularrotation seating” posture as an on-going process starting from “6. Inthe case of taking a nap”, the thinking process is in a recoveryconsciousness process even through the item is the same.

Note that the relationship information between the observable evaluationvalue and the recovery delay time of the driver currently driving maynot be sufficiently stored in the storage unit. In that case, forexample, recovery characteristic information generated on the basis ofinformation collected from driver population of the same age group isstored in the storage unit, and the recovery delay time t1 can becalculated using the recovery characteristic information as assumeddistribution information of recovery provided in advance. In therecovery information, the driver specific characteristics have notsufficiently been learned. Therefore, the same recovery ratio may beused on the basis of the information, or a higher recovery ratio may beset. Note that an ergonomically inexperienced user is expected torecover early in the beginning of use because the user is cautious.Then, the driver himself/herself adapts to the action in accordance withthe notification of the system as he/she gets accustomed to the system.Note that, in the case of using different vehicles in logistics businessthat operates many vehicles, in vehicle operation business that operatesbuses, taxis, or the like, or sharing cars and rental cars, personalauthentication of the driver is performed, the observable informationand recovery characteristics of driving are managed and learned in aconcentrated or distributed manner on a remote server or the like, andthe data of the recovery characteristics is not necessarily stored inthe individual vehicles and may be remotely learned and processed, andstored.

Furthermore, because the notification timing is important, the recoveryratio has been described using the uniform time to success or failure.However, the success or failure from the automatic driving to the manualdriving is not limited to the binary success or failure, anddetermination further extended to recovery takeover quality may be made.That is, delay time of recovery procedure transition to actual recoveryconfirmation, recovery start delay to the notification, stagnation in ahalfway recovery operation, and the like within allowed time may befurther input to the learning device as recovery quality evaluationvalues.

Referring back to FIG. 19, the description of the flow will becontinued. In step S44, a notification for prompting the driver torecover to driving is executed at the notification timing determinedaccording to the recovery delay time calculated in step S43, that is,timing when an event to be taken over (the takeover section from theautomatic driving to the manual driving or the cautioned travelingsection from the automatic driving) approaches the recovery delay time.This notification is executed as the display processing described abovewith reference to FIG. 2 or the display processing for the informationterminal 50 described with reference to FIG. 5, for example. Moreover,the notification may be executed as sound or alarm output, or vibrationof the steering wheel, the seat, the information terminal 50, or thelike. For example, in the case where the driver is taking a nap, anotification method for waking the driver from the sleeping state isselected.

As described above, the mobile device (automobile 10) determines thenotification timing of the manual driving recovery request notificationon the basis of the information of at least either the arousal level orthe position of the driver, and transmits the determined notificationtiming to the information terminal 50. Note that the notification timingmay be determined in consideration of a safety coefficient affected bythe state of the vehicle, the load, or the like. The informationterminal 50 executes the manual driving recovery request notification atthe notification timing received from the mobile device. That is, theinformation terminal 50 executes at least one of the display processing,the alarm output, or the vibration activation described with referenceto FIG. 5 at the notification timing determined by the mobile device.

Next, in step S45, the recovery transition of the driver is monitored.Then, in step S46, whether or not the driver can recover to drivingwithin the recovery delay time on the basis of a monitoring result instep S45. When it is determined that the driver can recover to driving,the driver recovers to driving in step S47. Then, in step S48, thelearning data is updated. That is, one sample value of the relationshipinformation (observation plot) between the observable evaluation valueand the actual recovery delay time regarding the initial type of thesecondary task of the driver when the above-described recovery todriving is performed is added. After that, the processing is terminated.Note that, in the present embodiment, the learning is limited to theplot data generated at each event. However, in reality, the learninglargely depends on the previous state (history) until the event occurs.Therefore, the estimation accuracy of the recovery delay required timefrom the observation value of the driver state may be further improvedby performing multidimensional learning.

Furthermore, when it is determined that recovery to driving is notpossible in step S46, a deceleration slowdown evacuation sequence isexecuted from the start to stop in step S51. Next, in step S52, a recordof penalty for occurrence of a takeover defect event is issued, and theprocessing is terminated. Note that the record of the penalty is storedin the storage unit.

As described above, in the case of a wristwatch-type device, forexample, the driver can always wear the device, and various advantagesare generated by providing the device with a driver state inputfunction. For example, history information of the driver's activity andbiometric information before boarding the vehicle can be acquired. Forexample, various types of prior information that may affect the drivingcharacteristics after boarding, such as fatigue status, breakacquisition history, and physical condition, can be acquired.Furthermore, in the user authentication, pulse waveforms or the like canbe used. Furthermore, as an advantage of the wearable terminal, thedriver's recognition for the notification information from the vehicleside system can be confirmed. The driver's recognition for thenotification and detection of a target action by the system areextremely important. In a case where the driver's action is notappropriate for the event notification to recover, the system needs toissue an alarm as a countermeasure for the delay in handling, causedeceleration or take an emergency action to gain time. The system needsto make a determination therefor. Furthermore, by providing a driver'saction detection function such as an accelerometer, a gyro sensor, or aposition detector, in addition to the detection function of the driver'sresponse input in the wearable device, the driver's behavior and statecan be observed, and whether or not the driver has responded orappropriately responded to the system notification can be detected.

[6. Configuration Example of Information Processing Device]

The above-described processing can be executed by applying theconfigurations of the mobile device 100 and the information terminal 50having the configuration described with reference to FIG. 8. Note thatpart of the processing can be executed by applying the configurations ofthe information processing device attachable to and detachable from themobile device and the information terminal 50, for example.

A hardware configuration example of the information terminal 50 and theinformation processing device attachable to and detachable from themobile device will be described with reference to FIG. 22.

FIG. 22 is a diagram illustrating an example of a hardware configurationapplicable as the information terminal 50 and the information processingdevice attachable to and detachable from the mobile device.

A central processing unit (CPU) 501 functions as a data processing unitthat execute various types of processing according to a program storedin a read only memory (ROM) 502 or a storage unit 508. For example, theCPU 501 executes processing according to the sequence described in theabove embodiment.

A random access memory (RAM) 503 stores the program executed by the CPU501, data, and the like. These CPU 501, ROM 502, and RAM 503 aremutually connected by a bus 504.

The CPU 501 is connected to an input/output interface 505 via the bus504. An input unit 506 including various switches, a keyboard, a touchpanel, a mouse, a microphone, and a state data acquisition unit such asa sensor, a camera, and GPS, and an output unit 507 including a display,a speaker, and the like are connected to the input/output interface 505.

Note that input information from a sensor 521 is also input to the inputunit 506.

Furthermore, the output unit 507 also outputs drive information for adrive unit 522 of the mobile device.

The CPU 501 receives commands, state data, and the like input from theinput unit 506, executes various types of information, and outputsprocessing results to the output unit 507, for example.

The storage unit 508 connected to the input/output interface 505includes, for example, a hard disk and the like, and stores the programexecuted by the CPU 501 and various data. A communication unit 509functions as a transmission/reception unit for data communication via anetwork such as the Internet or a local area network, and communicateswith an external device.

A drive 510 connected to the input/output interface 505 drives aremovable medium 511 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory such as a memory card,and executes data recording or reading.

[7. Conclusion of Configurations of Present Disclosure]

The examples of the present disclosure have been described in detailwith reference to the specific examples. However, it is obvious thatthose skilled in the art can make modifications and substitutions of theexamples without departing from the gist of the present disclosure. Thatis, the present invention has been disclosed in the form ofexemplification, and should not be restrictively interpreted. To judgethe gist of the present disclosure, the scope of claims should be takeninto consideration.

Note that the technology disclosed in the present specification can havethe following configurations.

(1) An information processing device that is a wearable terminalwearable on a human body, the information processing device including:

a display unit configured to output section information of an automaticdriving section and a manual driving section on a traveling route of amobile device, and display data by which a time to reach each section isconfirmable.

(2) The information processing device according to (1), in which thedisplay unit further outputs display data by which an output time of arecovery request notification to manual driving is confirmable.

(3) The information processing device according to (1) or (2), in whichthe display unit displays different pieces of the section informationaccording to individual time axes in an inner peripheral portion and anouter peripheral portion.

(4) The information processing device according to any one of (1) to(3), in which the display unit displays different pieces of the sectioninformation according to three or more individual time axes from aninner peripheral portion to an outer peripheral portion.

(5) The information processing device according to any one of (1) to(4), further including:

a sensor configured to detect a state of a terminal wearer; and

a communication unit configured to transmit detection information of thesensor to the mobile device.

(6) The information processing device according to any one of (1) to(5), further including:

a biosensor configured to detect a motion of a terminal wearer; and

a communication unit configured to transmit detection information of thebiosensor to the mobile device.

(7) The information processing device according to any one of (1) to(6), further including:

a position sensor configured to detect a position of a terminal wearer;and

a communication unit configured to transmit detection information of theposition sensor to the mobile device.

(8) The information processing device according to any one of (1) to(7), further including:

an activity amount measurement sensor configured to detect an activityamount of a terminal wearer; and

a communication unit configured to transmit detection information of theactivity amount measurement sensor to the mobile device.

(9) The information processing device according to any one of (1) to(8), further including:

a sensor configured to detect a state of a terminal wearer;

a storage unit configured to store acquired information of the sensor aslog data; and

a communication unit configured to transmit detection information of thesensor to the mobile device, in which

the log data indicating the state of the terminal wearer before theterminal wearer gets in the mobile device is output to the mobiledevice.

(10) The information processing device according to any one of (1) to(9), in which

the information processing device outputs section information indicatingeach section of the automatic driving section and the manual drivingsection on the traveling route of the mobile device, and a takeoversection set between the automatic driving section and the manual drivingsection, and the display data by which a time to reach each section isconfirmable.

(11) The information processing device according to any one of (1) to(10), in which

the information processing device has an alarm output function andconfigured to output an alarm at an output time of the recovery requestnotification to the manual driving.

(12) The information processing device according to any one of (1) to(11), in which

the information processing device has a vibration function andconfigured to perform vibration start processing at an output time ofthe recovery request notification to the manual driving.

(13) The information processing device according to any one of (1) to(12), in which

the information processing device is a wristwatch-type terminal.

(14) An information processing system including a mobile device and aninformation terminal having a wearable specification wearable on adriver of the mobile device, in which

the mobile device

executes processing of acquiring section information of an automaticdriving section and a manual driving section on a traveling route of themobile device, estimating a time to reach each section, and transmittingthe estimated time to the information terminal, and

the information terminal

receives transmission data from the mobile device, and

outputs the section information of an automatic driving section and amanual driving section on a traveling route of the mobile device, anddisplay data by which the time to reach each section is confirmable, toa display unit.

(15) The information processing system according to (14), in which

the mobile device acquires the section information of an automaticdriving section and a manual driving section from a local dynamic map(LDM).

(16) The information processing system according to (14) or (15), inwhich

the mobile device estimates the time to reach each section on the basisof average speed information of a vehicle on the traveling route of themobile device.

(17) The information processing system according to any one of (14) to(16), in which

the mobile device determines notification timing of a manual drivingrecovery request notification on the basis of at least either an arousallevel or a position of a driver, and transmits determined notificationtiming information to the information terminal, and

the information terminal executes the manual driving recovery requestnotification at the notification timing.

(18) The information processing system according to (17), in which theinformation terminal executes processing of at least either outputtingan alarm or starting vibration at the notification timing.

(19) An information processing method executed in an informationprocessing device,

the information processing device being a wearable terminal wearable ona human body, the information processing method including:

by a data processing unit, outputting section information of anautomatic driving section and a manual driving section on a travelingroute of a mobile device, and display data by which a time to reach eachsection is confirmable, to a display unit.

(20) An information processing method executed in an informationprocessing system including a mobile device and an information terminalhaving a wearable specification wearable on a driver of the mobiledevice, the information processing method including:

by the mobile device,

executing processing of acquiring section information of an automaticdriving section and a manual driving section on a traveling route of themobile device, estimating a time to reach each section, and transmittingthe estimated time to the information terminal; and

by the information terminal,

receiving transmission data from the mobile device; and

outputting the section information of an automatic driving section and amanual driving section on a traveling route of the mobile device, anddisplay data by which the time to reach each section is confirmable, toa display unit.

(21) A program for causing an information processing device to executeinformation processing,

the information processing device being a wearable terminal wearable ona human body,

the program for causing a data processing unit to output sectioninformation of an automatic driving section and a manual driving sectionon a traveling route of a mobile device, and display data by which atime to reach each section is confirmable, to a display unit.

Furthermore, the series of processing described in the description canbe executed by hardware, software, or a combined configuration of thehardware and software. In the case of executing the processing bysoftware, a program, in which the processing sequence is recorded, canbe installed in a memory of a computer incorporated in dedicatedhardware and executed by the computer, or the program can be installedin and executed by a general-purpose computer capable of executingvarious types of processing. For example, the program can be recorded inthe recording medium in advance. Other than the installation from therecording medium to the computer, the program can be received via anetwork such as a local area network (LAN) or the Internet and installedin a recording medium such as a built-in hard disk.

Note that the various types of processing described in the descriptionmay be executed not only in chronological order as described but also inparallel or individually depending on the processing capability of thedevice that executes the process or as required. Furthermore, the systemin the present description is a logical aggregate configuration of aplurality of devices, and is not limited to devices having respectiveconfigurations within the same housing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration of an embodiment of thepresent disclosure, a configuration to output section information of anautomatic driving section and a manual driving section and display databy which a time to reach each section is confirmable to a wearableterminal is implemented.

Specifically, for example, a mobile device acquires the sectioninformation of the automatic driving section and the manual drivingsection on a traveling route, and estimates the time to reach eachsection and transmits the estimated time to an information terminal. Theinformation terminal receives the transmission data from the mobiledevice, and outputs the section information of the automatic drivingsection and the manual driving section and the display data by which atime to reach each section is confirmable. Moreover, the mobile devicedetermines notification timing of a manual driving recovery requestnotification on the basis of an arousal level, a position of a driver,or the like, and transmits the determined notification timing to theinformation terminal, and the information terminal executes the manualdriving recovery request notification at the notification timing. In thecase of a constantly wearable information terminal, driver's historylogs before and after boarding, such as the driver's blood flow, can beacquired, and by installing a function to receive a health history suchas sleeping time on the mobile device, the arousal level can bedetermined with higher accuracy.

With the present configuration, the configuration to output sectioninformation of an automatic driving section and a manual driving sectionand display data by which a time to reach each section is confirmable toa wearable terminal is implemented.

REFERENCE SIGNS LIST

-   10 Automobile-   11 Data processing unit-   12 Driver information acquisition unit-   13 Environment information acquisition unit-   14 Communication unit-   15 Notification unit-   20 Driver-   30 Server-   50 Information terminal-   100 Mobile device-   101 Input unit-   102 Data acquisition unit-   103 Communication unit-   104 In-vehicle device-   105 Output control unit-   106 Output unit-   107 Drive system control unit-   108 Drive system-   109 Body system control unit-   110 Body system-   111 Storage unit-   112 Automatic driving control unit-   121 Communication network-   131 Detection unit-   132 Self-position estimation unit-   133 State analysis unit-   134 Planning unit-   135 Motion control unit-   141 Vehicle exterior information detection unit-   142 Vehicle interior information detection unit-   143 Vehicle state detection unit-   151 Map analysis unit-   152 Traffic rule recognition unit-   153 State recognition unit-   154 State prediction unit-   155 Safety determination unit (learning processing unit)-   161 Route planning unit-   162 Action planning unit-   163 Motion planning unit-   171 Emergency avoidance unit-   172 Acceleration and deceleration control unit-   173 Direction control unit-   501 CPU-   502 ROM-   503 RAM-   504 Bus-   505 Input/output interface-   506 Input unit-   507 Output unit-   508 Storage unit-   509 Communication unit-   510 Drive-   511 Removable medium-   521 Sensor-   522 Drive unit

1. An information processing device that is a wearable terminal wearableon a human body, the information processing device comprising: a displayunit configured to output section information of an automatic drivingsection and a manual driving section on a traveling route of a mobiledevice, and display data by which a time to reach each section isconfirmable.
 2. The information processing device according to claim 1,wherein the display unit further outputs display data by which an outputtime of a recovery request notification to manual driving isconfirmable.
 3. The information processing device according to claim 1,wherein the display unit displays different pieces of the sectioninformation according to individual time axes in an inner peripheralportion and an outer peripheral portion.
 4. The information processingdevice according to claim 1, wherein the display unit displays differentpieces of the section information according to three or more individualtime axes from an inner peripheral portion to an outer peripheralportion.
 5. The information processing device according to claim 1,further comprising: a sensor configured to detect a state of a terminalwearer; and a communication unit configured to transmit detectioninformation of the sensor to the mobile device.
 6. The informationprocessing device according to claim 1, further comprising: a biosensorconfigured to detect a motion of a terminal wearer; and a communicationunit configured to transmit detection information of the biosensor tothe mobile device.
 7. The information processing device according toclaim 1, further comprising: a position sensor configured to detect aposition of a terminal wearer; and a communication unit configured totransmit detection information of the position sensor to the mobiledevice.
 8. The information processing device according to claim 1,further comprising: an activity amount measurement sensor configured todetect an activity amount of a terminal wearer; and a communication unitconfigured to transmit detection information of the activity amountmeasurement sensor to the mobile device.
 9. The information processingdevice according to claim 1, further comprising: a sensor configured todetect a state of a terminal wearer; a storage unit configured to storeacquired information of the sensor as log data; and a communication unitconfigured to transmit detection information of the sensor to the mobiledevice, wherein the log data indicating the state of the terminal wearerbefore the terminal wearer gets in the mobile device is output to themobile device.
 10. The information processing device according to claim1, wherein the information processing device outputs section informationindicating each section of the automatic driving section and the manualdriving section on the traveling route of the mobile device, and atakeover section set between the automatic driving section and themanual driving section, and the display data by which a time to reacheach section is confirmable.
 11. The information processing deviceaccording to claim 1, wherein the information processing device has analarm output function and configured to output an alarm at an outputtime of the recovery request notification to the manual driving.
 12. Theinformation processing device according to claim 1, wherein theinformation processing device has a vibration function and configured toperform vibration start processing at an output time of the recoveryrequest notification to the manual driving.
 13. The informationprocessing device according to claim 1, wherein the informationprocessing device is a wristwatch-type terminal.
 14. An informationprocessing system comprising a mobile device and an information terminalhaving a wearable specification wearable on a driver of the mobiledevice, wherein the mobile device executes processing of acquiringsection information of an automatic driving section and a manual drivingsection on a traveling route of the mobile device, estimating a time toreach each section, and transmitting the estimated time to theinformation terminal, and the information terminal receives transmissiondata from the mobile device, and outputs the section information of anautomatic driving section and a manual driving section on a travelingroute of the mobile device, and display data by which the time to reacheach section is confirmable, to a display unit.
 15. The informationprocessing system according to claim 14, wherein the mobile deviceacquires the section information of an automatic driving section and amanual driving section from a local dynamic map (LDM).
 16. Theinformation processing system according to claim 14, wherein the mobiledevice estimates the time to reach each section on a basis of averagespeed information of a vehicle on the traveling route of the mobiledevice.
 17. The information processing system according to claim 14,wherein the mobile device determines notification timing of a manualdriving recovery request notification on a basis of at least either anarousal level or a position of a driver, and transmits determinednotification timing information to the information terminal, and theinformation terminal executes the manual driving recovery requestnotification at the notification timing.
 18. The information processingsystem according to claim 17, wherein the information terminal executesprocessing of at least either outputting an alarm or starting vibrationat the notification timing.
 19. An information processing methodexecuted in an information processing device, the information processingdevice being a wearable terminal wearable on a human body, theinformation processing method comprising: by a data processing unit,outputting section information of an automatic driving section and amanual driving section on a traveling route of a mobile device, anddisplay data by which a time to reach each section is confirmable, to adisplay unit.
 20. An information processing method executed in aninformation processing system including a mobile device and aninformation terminal having a wearable specification wearable on adriver of the mobile device, the information processing methodcomprising: by the mobile device, executing processing of acquiringsection information of an automatic driving section and a manual drivingsection on a traveling route of the mobile device, estimating a time toreach each section, and transmitting the estimated time to theinformation terminal; and by the information terminal, receivingtransmission data from the mobile device; and outputting the sectioninformation of an automatic driving section and a manual driving sectionon a traveling route of the mobile device, and display data by which thetime to reach each section is confirmable, to a display unit.
 21. Aprogram for causing an information processing device to executeinformation processing, the information processing device being awearable terminal wearable on a human body, the program for causing adata processing unit to output section information of an automaticdriving section and a manual driving section on a traveling route of amobile device, and display data by which a time to reach each section isconfirmable, to a display unit.